Remote Control Operations in a Video Conference

ABSTRACT

Some embodiments provide a method for allowing a first device that is in a video conference with a second mobile device to remotely control the second mobile device. The method sends images captured by a camera of the first device to the second device. The method receives images captured by a camera of the second device. The method sends a command through a communication channel of a real-time communication session to the second device. The command is for instructing the second device to perform an operation that modifies the images captured by the camera of the second device.

BACKGROUND

Many of today's portable devices, such as smartphones, provide videocapture functionality. A user of the portable device can capture bothstill images and video through a camera on the phone. However, totransmit captured video to another party, the user must generally eithersend the video directly to the other party or upload the video toanother location (e.g., an Internet video hosting site) after the videois done being captured. Unfortunately, this does not allow the otherparty to view the live video stream as it is captured by the portabledevice.

In addition, standard portable devices are only equipped with onecamera, and processing information from this one camera is difficultenough. An ideal device would have multiple cameras and could send outlive video that is a composition of video from at least two cameras.This is an especially difficult problem in light of the limitedresources available for portable devices, both in terms of the deviceprocessing multiple captured video streams and a network to which thedevice is connected handling the transmission of the live video streams.

BRIEF SUMMARY

Some embodiments of the invention provide a mobile device with twocameras that can take pictures and videos. The mobile device of someembodiments has a display screen for displaying the captured pictureimages and video images. It also includes a storage for storing thecaptured images for later transmission to another device. The devicefurther has a network interface that allows the device to transmit thecaptured images to one or more devices during a real-time communicationsession between the users of the devices. The device also includes anencoder that it can use to encode the captured images for local storageor for transmission to another device. The mobile device furtherincludes a decoder that allows the device to decode images captured byanother device during a real-time communication session or to decodeimages stored locally.

One example of a real-time communication session that involves thetransmission of the captured video images is a video conference. In someembodiments, the mobile device can only transmit one camera's capturedvideo images at any given time during a video conference. In otherembodiments, however, the mobile device can transmit captured videoimages from both of its cameras simultaneously during a video conferenceor other real-time communication session.

During a video conference with another device, the mobile device of someembodiments can transmit other types of content along with the videocaptured by one or both of its cameras. One example of such othercontent includes low or high resolution picture images that are capturedby one of the device's cameras, while the device's other camera iscapturing a video that is used in the video conference. Other examplesof such other content include (1) files and other content stored on thedevice, (2) the screen display of the device (i.e., the content that isdisplayed on the device's screen), (3) content received from anotherdevice during a video conference or other real-time communicationsession, etc.

The mobile devices of some embodiments employ novel in-conferenceadjustment techniques for making adjustments during a video conference.For instance, while transmitting only one camera's captured video duringa video conference, the mobile device of some embodiments candynamically switch to transmitting a video captured by its other camera.In such situations, the mobile device of some embodiments notifies anyother device participating in the video conference of this switch sothat this other device can provide a smooth transition on its endbetween the videos captured by the two cameras.

In some embodiments, the request to switch cameras not only canoriginate on the “local” device that switches between its cameras duringthe video conference, but also can originate from the other “remote”device that is receiving the video captured by the local device.Moreover, allowing one device to direct another device to switch camerasis just one example of a remote control capability of the devices ofsome embodiments. Examples of other operations that can be directed to adevice remotely in some embodiments include exposure adjustmentoperations (e.g., auto-exposure), focus adjustment operations (e.g.,auto-focus), etc. Another example of a novel in-conference adjustmentthat can be specified locally or remotely is the identification of aregion of interest (ROI) in a captured video, and the use of this ROIidentification to modify the behavior of the capturing camera, to modifythe image processing operation of the device with the capturing camera,or to modify the encoding operation of the device with the capturingcamera.

Yet another example of a novel in-conference adjustment of someembodiments involves real-time modifications of composite video displaysthat are generated by the devices. Specifically, in some embodiments,the mobile devices generate composite displays that simultaneouslydisplay multiple videos captured by multiple cameras of one or moredevices. In some cases, the composite displays place the videos inadjacent display areas (e.g., in adjacent windows). In other cases, thecomposite display is a picture-in-picture (PIP) display that includes atleast two display areas that show two different videos where one of thedisplay areas is a background main display area and the other is aforeground inset display area that overlaps the background main displayarea.

The real-time modifications of the composite video displays in someembodiments involve moving one or more of the display areas within acomposite display in response to a user's selection and movement of thedisplay areas. Some embodiments also rotate the composite display duringa video conference, when the screen of the device that provides thiscomposite display rotates. Also, the mobile device of some embodimentsallows the user of the device to swap the videos in a PIP display (i.e.,to make the video in the foreground inset display appear in thebackground main display while making the video in the background maindisplay appear in the foreground inset display).

The preceding Summary is intended to serve as a brief introduction tosome embodiments of the invention. It is not meant to be an introductionor overview of all inventive subject matter disclosed in this document.The Detailed Description that follows and the Drawings that are referredto in the Detailed Description will further describe the embodimentsdescribed in the Summary as well as other embodiments. Accordingly, tounderstand all the embodiments described by this document, a full reviewof the Summary, Detailed Description and the Drawings is needed.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appendedclaims. However, for purpose of explanation, several embodiments of theinvention are set forth in the following figures.

FIG. 1 illustrates a composite display of some embodiments.

FIG. 2 illustrates another composite display of some embodiments.

FIG. 3 conceptually illustrates a software architecture for a videoprocessing and encoding module of a dual camera mobile device of someembodiments.

FIG. 4 conceptually illustrates a captured image processing unit of someembodiments.

FIG. 5 conceptually illustrates examples of different frame rates basedon different vertical blanking intervals (VBIs).

FIG. 6 conceptually illustrates a software architecture for a videoconferencing and processing module of a dual camera mobile device ofsome embodiments.

FIG. 7 conceptually illustrates an example video conference requestmessaging sequence of some embodiments.

FIG. 8 illustrates a user interface of some embodiments for a videoconference setup operation.

FIG. 9 illustrates a user interface of some embodiments for accepting aninvitation to a video conference.

FIG. 10 illustrates another user interface of some embodiments foraccepting an invitation to a video conference.

FIG. 11 illustrates another user interface of some embodiments for avideo conference setup operation.

FIG. 12 conceptually illustrates another software architecture for avideo conferencing and processing module of a dual camera mobile deviceof some embodiments.

FIG. 13 conceptually illustrates another software architecture for adual camera mobile device of some embodiments.

FIG. 14 conceptually illustrates a process performed by a videoconference manager of some embodiments such as that illustrated in FIG.12.

FIG. 15 conceptually illustrates a process performed by an imageprocessing manager of some embodiments such as that illustrated in FIG.6.

FIG. 16 illustrates a user interface of some embodiments for an exposureadjustment operation.

FIG. 17 illustrates a user interface of some embodiments for a focusadjustment operation.

FIG. 18 conceptually illustrates a software architecture for anetworking manager of some embodiments such as that illustrated in FIG.12.

FIG. 19 illustrates a user interface of some embodiments for a PIPdisplay rotation operation.

FIG. 20 illustrates another user interface of some embodiments for a PIPdisplay rotation operation.

FIG. 21 illustrates another user interface of some embodiments for a PIPdisplay rotation operation.

FIG. 22 illustrates another user interface of some embodiments for a PIPdisplay rotation operation.

FIG. 23 illustrates a user interface of some embodiments for identifyinga region of interest in a display.

FIG. 24 illustrates another user interface of some embodiments foridentifying a region of interest in a display.

FIG. 25 illustrates another user interface of some embodiments foridentifying a region of interest in a display.

FIG. 26 illustrates a process of some embodiments for performing a localswitch camera operation on a dual camera mobile device.

FIG. 27 illustrates a user interface of some embodiments for a switchcamera operation.

FIG. 28 illustrates another user interface of some embodiments for aswitch camera operation.

FIG. 29 illustrates another user interface of some embodiments for aswitch camera operation.

FIG. 30 illustrates another user interface of some embodiments for aswitch camera operation.

FIG. 31 illustrates a process of some embodiments for performing aremote switch camera operation on a dual camera mobile device.

FIG. 32 illustrates a user interface of some embodiments for a remotecontrol switch camera operation.

FIG. 33 illustrates another user interface of some embodiments for aremote control switch camera operation.

FIG. 34 illustrates another user interface of some embodiments for aremote control switch camera operation.

FIG. 35 illustrates another user interface of some embodiments for aremote control switch camera operation.

FIG. 36 conceptually illustrates a process of some embodiments forperforming an exposure adjustment operation.

FIG. 37 illustrates a user interface of some embodiments for an exposureadjustment operation.

FIG. 38 illustrates another user interface of some embodiments for anexposure adjustment operation.

FIG. 39 illustrates another user interface of some embodiments for anexposure adjustment operation.

FIG. 40 conceptually illustrates an exposure adjustment processperformed by an image processing manager of some embodiments such asthat illustrated in FIG. 12.

FIG. 41 conceptually illustrates exposure adjustment operations of someembodiments.

FIG. 42 conceptually illustrates a process of some embodiments forperforming a focus adjustment operation.

FIG. 43 illustrates a user interface of some embodiments for a focusadjustment operation.

FIG. 44 illustrates another user interface of some embodiments for afocus adjustment operation.

FIG. 45 illustrates another user interface of some embodiments for afocus adjustment operation.

FIG. 46 conceptually illustrates an application programming interface(API) architecture of some embodiments.

FIG. 47 illustrates an architecture for a dual camera mobile computingdevice of some embodiments.

FIG. 48 conceptually illustrates a touch input/output (I/O) device ofsome embodiments.

FIG. 49 conceptually illustrates an example communication system of someembodiments.

FIG. 50 conceptually illustrates another example communication system ofsome embodiments.

DETAILED DESCRIPTION

In the following description, numerous details are set forth for purposeof explanation. However, one of ordinary skill in the art will realizethat the invention may be practiced without the use of these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order not to obscure the description of theinvention with unnecessary detail.

Some embodiments of the invention provide a mobile device with twocameras that can take pictures and videos. Examples of mobile devicesinclude mobile phones, smartphones, personal digital assistants (PDAs),laptops, tablet personal computers, or any other type of mobilecomputing device. As used in this document, pictures refer to stillpicture images that are taken by the camera one at a time in asingle-picture mode, or several at a time in a fast-action mode. Video,on the other hand, refers to a sequence of video images that arecaptured by a camera at a particular rate, which is often referred to asa frame rate. Typical frame rates for capturing video are 25 frames persecond (fps), 30 fps, and 60 fps. The cameras of the mobile device ofsome embodiments can capture video images (i.e., video frames) at theseand other frame rates.

The mobile device of some embodiments (1) can display the capturedpicture images and video images, (2) can store the captured images forlater transmission to another device, (3) can transmit the capturedimages to one or more devices during a real-time communication sessionbetween the users of the devices, and (4) can encode the captured imagesfor local storage or for transmission to another device.

One example of a real-time communication session that involves thetransmission of the captured video images is a video conference. In someembodiments, the mobile device can only transmit one camera's capturedvideo images at any given time during a video conference. In otherembodiments, however, the mobile device can transmit captured videoimages from both of its cameras simultaneously during a video conferenceor other real-time communication session.

The mobile devices of some embodiments generate composite displays thatinclude simultaneous display of multiple videos captured by multiplecameras of one or more devices. In some cases, the composite displaysplace the videos in adjacent display areas (e.g., in adjacent windows).FIG. 1 illustrates one such example of a composite display 100 thatincludes two adjacent display areas 105 and 110 that simultaneouslydisplay two videos captured by two cameras of one device or captured bytwo cameras of two different devices that are in a video conference.

In other cases, the composite display is a PIP display that includes atleast two display areas that show two different videos, where one of thedisplay areas is a background main display area and the other is aforeground inset display area that overlaps the background main displayarea. FIG. 2 illustrates one such example of a composite PIP display200. This composite PIP display 200 includes a background main displayarea 205 and a foreground inset display area 210 that overlaps thebackground main display area. The two display areas 205 and 210simultaneously display two videos captured by two cameras of one device,or captured by two cameras of two different devices that are in a videoconference. While the example composite PIP displays illustrated anddiscussed in this document are similar to the composite PIP display 200,which shows the entire foreground inset display area 210 within thebackground main display area 205, other composite PIP displays that havethe foreground inset display area 210 overlapping, but not entirelyinside, the background main display area 205 are possible.

In addition to transmitting video content during a video conference withanother device, the mobile device of some embodiments can transmit othertypes of content along with the conference's video content. One exampleof such other content includes low or high resolution picture imagesthat are captured by one of the device's cameras, while the device'sother camera is capturing a video that is used in the video conference.Other examples of such other content include (1) files and other contentstored on the device, (2) the screen display of the device (i.e., thecontent that is displayed on the device's screen), (3) content receivedfrom another device during a video conference or other real-timecommunication session, etc.

The mobile devices of some embodiments employ novel in-conferenceadjustment techniques for making adjustments during a video conference.For instance, while transmitting only one camera's captured video duringa video conference, the mobile device of some embodiments candynamically switch to transmitting the video captured by its othercamera. In such situations, the mobile device of some embodimentsnotifies any other device participating in the video conference of thisswitch so that this other device can provide a smooth transition on itsend between the videos captured by the two cameras.

In some embodiments, the request to switch cameras not only canoriginate on the “local” device that switches between its cameras duringthe video conference, but also can originate from the other “remote”device that is receiving the video captured by the local device.Moreover, allowing one device to direct another device to switch camerasis just one example of a remote control capability of the devices ofsome embodiments. Examples of other operations that can be directed to adevice remotely in some embodiments include exposure adjustmentoperations (e.g., auto-exposure), focus adjustment operations (e.g.,auto-focus), etc. Another example of a novel in-conference adjustmentthat can be specified locally or remotely is the identification of aregion of interest (ROI) in a captured video, and the use of this ROIidentification to modify the behavior of the capturing camera, to modifythe image processing operation of the device with the capturing camera,or to modify the encoding operation of the device with the capturingcamera.

Yet another example of a novel in-conference adjustment of someembodiments involves real-time modifications of composite video displaysthat are generated by the devices. Specifically, in some embodiments,the real-time modifications of the composite video displays involvemoving one or more of the display areas within a composite display inresponse to a user's selection and movement of the display areas. Someembodiments also rotate the composite display during a video conference,when the screen of the device that provides this composite displayrotates. Also, the mobile device of some embodiments allow the user ofthe device to flip the order of videos in a PIP display (i.e., to makethe video in the foreground inset display appear in the background maindisplay, while making the video in the background main display appear inthe foreground inset display).

Several more detailed embodiments are described below. Section Iprovides a description of the video processing architecture of someembodiments. Section II then describes the captured image processingunit of some embodiments. In some embodiments, this unit is thecomponent of the device that is responsible for processing raw imagescaptured by the cameras of the device.

Next, Section III describes the video conferencing architecture of someembodiments. This section also describes the video conference module ofsome embodiments, as well as several manners for setting up a singlecamera video conference. Section IV then describes in-conferenceadjustment and control operations of some embodiments. Section V nextdescribes the hardware architecture of the dual camera device of someembodiments. Lastly, U.S. patent application Ser. No. ______, entitled“Establishing a Video Conference During a Phone Call,” filedconcurrently with the present application, with the Attorney Docket No.APLE.P0212, describes several additional embodiments relating to some ofthe features described above, such as some of the in-conferenceadjustments, etc. This U.S. patent application entitled “Establishing aVideo Conference During a Phone Call” is incorporated herein byreference.

I. Video Capture and Processing

FIG. 3 conceptually illustrates a video processing and encoding module300 of a dual camera mobile device of some embodiments. In someembodiments, the module 300 processes images and encodes videos that arecaptured by the cameras of the dual camera mobile device. As shown inFIG. 3, this module 300 includes a captured image processing unit (CIPU)driver 305, a media exchange module 310, an encoder driver 320, and avideo processing module 325.

In some embodiments, the media exchange module 310 allows programs onthe device that are consumers and producers of media content to exchangemedia content and instructions regarding the processing of the mediacontent. In the video processing and encoding module 300, the mediaexchange module 310 of some embodiments routes instructions and mediacontent between the video processing module 325 and the CIPU driver 305,and between the video processing module 325 and the encoder driver 320.To facilitate the routing of such instructions and media content, themedia exchange module 310 of some embodiments provides a set ofapplication programming interfaces (APIs) for the consumers andproducers of media content to use. In some of such embodiments, themedia exchange module 310 is a set of one or more frameworks that ispart of an operating system running on the dual camera mobile device.One example of such a media exchange module 310 is the Core Mediaframework provided by Apple Inc.

The video processing module 325 performs image processing on the imagesand/or the videos captured by the cameras of the device. Examples ofsuch operations include exposure adjustment operations, focus adjustmentoperations, perspective correction, dynamic range adjustment, imageresizing, image compositing, etc. In some embodiments, some imageprocessing operations can also be performed by the media exchange module310. For instance, as shown in FIG. 3, the media exchange module 310 ofsome embodiments performs a temporal noise reduction (TNR) operation(e.g., by TNR 315) that reduces noise in video images captured by thecameras of the device. Further examples of such image processingoperations of the video processing module 325 and the media exchangemodule 310 will be provided below.

Through the media exchange module 310, the video processing module 325interfaces with the CIPU driver 305 and the encoder driver 320, asmentioned above. The CIPU driver 305 serves as a communication interfacebetween a captured image processing unit (CIPU) 330 and the mediaexchange module 310. As further described below, the CIPU 330 is thecomponent of the dual camera device that is responsible for processingimages captured during image capture or video capture operations of thedevice's cameras. From the video processing module 325 through the mediaexchange module 310, the CIPU driver 305 receives requests for imagesand/or videos from one or both of the device's cameras. The CIPU driver305 relays such requests to the CIPU 330, and in response receives therequested images and/or videos from the CIPU 330, which the CIPU driver305 then sends to the video processing module 325 through the mediaexchange module 310. Through the CIPU driver 305 and the media exchangemodule 310, the video processing module 325 of some embodiments alsosends instructions to the CIPU 330 in order to modify some of itsoperations (e.g., to modify a camera's frame rate, exposure adjustmentoperation, focus adjustment operation, etc.).

The encoder driver 320 serves as a communication interface between themedia exchange module 310 and an encoder hardware 335 (e.g., an encoderchip, an encoding component on a system on chip, etc.). In someembodiments, the encoder driver 320 receives images and requests toencode the images from the video processing module 325 through the mediaexchange module 310. The encoder driver 320 sends the images to beencoded to the encoder 335, which then performs picture encoding orvideo encoding on the images. When the encoder driver 320 receivesencoded images from the encoder 335, the encoder driver 320 sends theencoded images back to the video processing module 325 through the mediaexchange module 310.

In some embodiments, the video processing module 325 can performdifferent operations on the encoded images that it receives from theencoder. Examples of such operations include storing the encoded imagesin a storage of the device, transmitting the encoded images in a videoconference through a network interface of the device, etc.

In some embodiments, some or all of the modules of the video processingand encoding module 300 are implemented as part of an operating system.For example, some embodiments implement all four components 305, 310,320, and 325 of this module 300 as part of the operating system of thedevice. Other embodiments implement the media exchange module 310, theCIPU driver 305, and the encoder driver 320 as part of the operatingsystem of the device, while having the video processing module 325 as anapplication that runs on the operating system. Still, otherimplementations of the module 300 are possible.

The operation of the video processing and encoding module 300 during avideo capture session will now be described. To start a video capturesession, the video processing module 325 initializes several componentsthat are needed for the video capture session. In some embodiments,these components include (1) the CIPU 330, (2) a scaling and compositingmodule (not shown) of the video processing module 325, (3) an imageprocessing module (not shown) of the video processing module 325, and(4) the encoder 335. Also, the video processing module 325 of someembodiments initializes a network manager (not shown) when it isparticipating in a video conference.

Through the media exchange module 310 and the CIPU driver 305, the videoprocessing module sends its initialization request to the CIPU 330, inorder to have one or both of the cameras of the device start videocapturing. In some embodiments, this request specifies a particularframe rate, exposure level, and scaling size for each camera that needsto capture a video. In response to this request, the CIPU 330 starts toreturn video images from the requested cameras at the specified rate(s),exposure level(s), and scaling size(s). These video images are returnedto the video processing module 325 through the CIPU driver 305 and themedia exchange module 310, which, as mentioned above, performs TNRoperations on the video images before supplying them to the videoprocessing module 325. At the video processing module 325, the videoimages are stored in a buffer (not shown) for additional imageprocessing.

The image processing module of the video processing module 325 retrievesthe video images stored in the buffer for additional video processing.The scaling and compositing module then retrieves the processed videoimages in order to scale them if necessary for real time display on thedisplay screen of the device. In some embodiments, this module createscomposite images from the images captured by two cameras of the deviceor from images captured by the camera(s) of the device along with thecamera(s) of another device during a video conference in order toprovide a real-time display of the captured video images on the deviceor to create a composite video image for encoding.

The processed and/or composited video images are supplied to the encoder335 through the encoder driver 320 and the media exchange module 310.The encoder 335 then encodes the video images. The encoded images arethen returned to the video processing module 325 (again through theencoder driver 320 and the media exchange module 310) for storage on thedevice or for transmission during a video conference. When the device isparticipating in a video conference, the network manager (that wasinitialized by the video processing module 325) then retrieves theseencoded images, packetizes them and transmits them to one or more otherdevices through a network interface (not shown) of the device.

II. Captured Image Processing

The images captured by cameras of the dual camera mobile device of someembodiments are raw, unprocessed images. These images require conversionto a particular color space before the images can be used for otheroperations such as transmitting the images to another device (e.g.,during a video conference), storing the images, or displaying theimages. In addition, the images captured by the cameras may need to beprocessed to correct errors and/or distortions and to adjust the images'color, size, etc. Accordingly, some embodiments perform severalprocessing operations on the images before storing, transmitting, anddisplaying such images. Part of the processing of such images isperformed by the CIPU 330.

One example of such a CIPU is illustrated in FIG. 4. Specifically, thisfigure conceptually illustrates a captured image processing unit (CIPU)400 of some embodiments. This CIPU 400 includes a single processingpipeline 485 that either processes images from only one of the device'scameras at a time, or processes images from both of the device's camerassimultaneously in a time-division multiplex fashion (i.e., in a timeinterleaved manner). The CIPU 400's processing pipeline 485 can beconfigured differently to address differing characteristics and/oroperational settings of the different cameras. Examples of differentcamera characteristics in some embodiments include differentresolutions, noise sensors, lens types (fixed or zoom lens), etc. Also,examples of different operational settings under which the device canoperate the cameras in some embodiments include image resolution size,frame rate, zoom level, exposure level, etc.

As shown in FIG. 4, the CIPU 400 includes a sensor module 415, aline/frame buffer 417, a bad pixel correction (BPC) module 420, a lensshading (LS) module 425, a demosaicing module 430, a white balance (WB)module 435, a gamma module 440, a color space conversion (CSC) module445, a hue, saturation, and contrast (HSC) module 450, a scaler module455, a filter module 460, a statistics engine 465, two sets of registers470, and a controller module 475. In some embodiments, all of themodules of the CIPU 400 are implemented in hardware (e.g., an ASIC,FPGA, a SOC with a microcontroller, etc.), while in other embodiments,some or all of the modules of the CIPU 400 are implemented in software.

As shown in FIG. 4, the sensor module 415 communicatively couples to twopixel arrays 410 a and 410 b and two sets of sensors 405 a and 405 b oftwo cameras of the device. In some embodiments, this communicativecoupling is facilitated through each camera sensor's mobile industryprocessor interface (MIPI).

Through this communicative coupling, the sensor module 415 can forwardinstructions to the cameras to control various aspects of each camera'soperations such as its power level, zoom level, focus, exposure level,etc. In some embodiments, each camera has four operational power modes.In the first operational power mode, the camera is powered off. For thesecond operational power mode, the camera is powered on, but it is notyet configured. In the third operational power mode, the camera ispowered on, the camera's sensor is configured, and the camera sensor'spixels are collecting photons and converting the collected photons todigital values. However, the camera sensor is not yet sending images tothe sensor module 415. Finally, in the fourth operational power mode,the camera is in the same operational power mode as the third power modeexcept the camera is now sending images to the sensor module 415.

During the operation of the device, the cameras may switch from oneoperational power mode to another any number of times. When switchingoperational power modes, some embodiments require the cameras to switchoperational power modes in the order described above. Therefore, inthose embodiments, a camera in the first operational power mode can onlyswitch to the second operational power mode. When the camera is in thesecond operational power mode, it can switch to the first operationalpower mode or to the third operational power mode. Similarly, the cameracan switch from the third operational power mode to the secondoperational power mode or the fourth operation power mode. When thecamera is in the fourth operational power mode, it can only switch backto the third operational power mode.

Moreover, switching from one operational power mode to the next or theprevious operational power mode takes a particular amount of time. Thus,switching between two or three operational power modes is slower thanswitching between one operational power mode. The different operationalpower modes also consume different amounts of power. For instance, thefourth operational power mode consumes the most amount of power, thethird operational power mode consumes more power than the first andsecond, and the second operational power mode consumes more than thefirst. In some embodiments, the first operational power mode does notconsume any power.

When a camera is not in the fourth operational power mode capturingimages, the camera may be left in one of the other operational powermodes. Determining the operational mode in which to leave the unusedcamera depends on how much power the camera is allowed to consume andhow fast the camera may need to respond to a request to start capturingimages. For example, a camera configured to operate in the thirdoperational power mode (e.g., standby mode) consumes more power than acamera configured to be in the first operational power mode (i.e.,powered off). However, when the camera is instructed to capture images,the camera operating in the third operational power mode can switch tothe fourth operational power mode faster than the camera operating inthe first operational power mode. As such, the cameras can be configuredto operate in the different operational power modes when not capturingimages based on different requirements (e.g., response time to a requestto capture images, power consumption).

Through its communicative coupling with each camera, the sensor module415 can direct one or both sets of camera sensors to start capturingimages when the video processing module 325 requests one or both camerasto start capturing images and the sensor module 415 receives thisrequest through the controller module 475, as further described below.Bayer filters are superimposed over each of the camera sensors and thuseach camera sensor outputs Bayer pattern images, which are stored in thepixel array associated with each camera sensor. A Bayer pattern image isan image where each pixel only stores one color value: red, blue, orgreen.

Through its coupling with the pixel arrays 410 a and 410 b, the sensormodule 415 retrieves raw Bayer pattern images stored in the camera pixelarrays 410 a and 410 b. By controlling the rate at which the sensormodule 415 retrieves images from a camera's pixel array, the sensormodule 415 can control the frame rate of the video images that are beingcaptured by a particular camera. By controlling the rate of its imageretrieval, the sensor module 415 can also interleave the fetching ofimages captured by the different cameras in order to interleave the CIPUprocessing pipeline 485's image processing of the captured images fromthe different cameras. The sensor module 415's control of its imageretrieval is further described below and in the above-incorporated U.S.patent application Ser. No. ______, entitled “Establishing VideoConference During a Phone Call,” with Attorney Docket Number APLE.P0212.

The sensor module 415 stores image lines (i.e., rows of pixels of animage) in the line/frame buffer 417, which the sensor module 415retrieves from the pixel arrays 410 a and 410 b. Each image line in theline/frame buffer 417 is processed through the CIPU processing pipeline485. As shown in FIG. 4, the CIPU processing pipeline 485 is formed bythe BPC module 420, the LS module 425, the demosaicing module 430, theWB module 435, the gamma module 440, the CSC module 445, the HSC module450, the scaler module 455, and the filter module 460. In someembodiments, the CIPU processing pipeline 485 processes images from theline/frame buffer 417 on a line-by-line (i.e., row-by-row) basis whilein other embodiments the CIPU processing pipeline 485 processes entireimages from the line/frame buffer 417 on a frame-by-frame basis.

In the exemplary pipeline illustrated in FIG. 4, the BPC module 420 isthe module that retrieves the images from the line/frame buffer 417.This module performs a bad-pixel removal operation that attempts tocorrect bad pixels in the retrieved images that might have resulted fromone or more of the camera sensors being defective (e.g., the defectivephoto sensors do not sense light at all, sense light incorrectly, etc.).In some embodiments, the BPC module 420 detects bad pixels by comparinga particular pixel in an image with one or more neighboring pixels inthe image. If the difference between the value of the particular pixeland the values of the neighboring pixels is greater than a thresholdamount, the particular pixel's value is replaced by the average ofseveral neighboring pixel's values that are of the same color (i.e.,red, green, and blue) as the particular pixel.

The operation of the BPC module 420 is in part controlled by the valuesstored for this module in the two sets of registers 470 of the CIPU 400.Specifically, to process the images captured by the two differentcameras of the device, some embodiments configure the CIPU processingpipeline 485 differently for each camera, as mentioned above. The CIPUprocessing pipeline 485 is configured for the two different cameras bystoring two different sets of values in the two different sets ofregisters 470 a (Ra) and 470 b (Rb) of the CIPU 400. Each set ofregisters 470 includes one register (Ra or Rb) for each of the modules420-460 within the CIPU processing pipeline 485. Each register in eachregister set stores a set of values that defines one processing pipelinemodule's operation. Accordingly, as shown in FIG. 4, the register set470 a is for indicating the mode of operation of each processingpipeline module for one camera (camera A) of the dual camera mobiledevice, while the register set 470 b is for indicating the mode ofoperation of each module for the other camera (camera B) of the dualcamera mobile device.

One example of configuring the CIPU processing pipeline 485 differentlyfor each camera is to configure the modules of the CIPU processingpipeline 485 to process different sized images. For instance, if thecamera sensor 405 a is 640×480 pixels and the camera sensor 405 b is2048×1536 pixels, the set of registers 470 a is configured to storevalues that instruct the modules of the CIPU processing pipeline 485 toprocess 640×480 pixel images and the set of registers 470 b isconfigured to store values that instruct the modules of the CIPUprocessing pipeline 485 to process 2048×1536 pixel images.

In some embodiments, different processing pipeline configurations (i.e.,register values) are stored in different profile settings. In some ofsuch embodiments, a user of the mobile device is allowed to select oneof the profile settings (e.g., through a user interface displayed on themobile device) to set the operation of a camera(s). For example, theuser may select a profile setting for configuring a camera to capturehigh resolution video, a profile setting for configuring the same camerato capture low resolution video, or a profile setting for configuringboth cameras to capture high resolution still images. Differentconfigurations are possible, which can be stored in many differentprofile settings. In other of such embodiments, instead of allowing theuser to select a profile setting, a profile setting is automaticallyselected based on which application or activity the user selects. Forinstance, if the user selects a video conferencing application, aprofile that configures both cameras to capture video is automaticallyselected, if the user selects a photo application, a profile thatconfigures one of the cameras to capture still images is automaticallyselected, etc.

After the BPC module 420, the LS module 425 receives thebad-pixel-corrected images. The LS module 425 performs a lens shadingcorrection operation to correct for image defects that are caused bycamera lenses that produce light falloff effects (i.e., light is reducedtowards the edges of the camera sensor). Such effects cause images to beunevenly illuminated (e.g., darker at corners and/or edges). To correctthese image defects, the LS module 425 of some embodiments estimates amathematical model of a lens' illumination fall-off. The estimated modelis then used to compensate the lens fall-off of the image to evenlyilluminate unevenly illuminated portions of the image. For example, if acorner of the image is half the brightness of the center of the image,the LS module 425 of some embodiments multiplies the corner pixels valueby two in order to produce an even image.

The demosaicing module 430 performs a demosaicing operation to generatefull color images from images of sampled colors. As noted above, thecamera sensors output Bayer pattern images, which are incomplete becauseeach pixel of a Bayer pattern image stores only one color value. Thedemosaicing module 430 reconstructs a red, green, blue (RGB) image froma Bayer pattern image by interpolating the color values for each set ofcolors in the Bayer pattern image.

The WB module 435 performs a white balance operation on the RGB imagesreceived from the demosaicing module 430 so that the colors of thecontent of the images are similar to the colors of such contentperceived by the human eye in real life. The WB module 435 adjusts thewhite balance by adjusting colors of the images to render neutral colors(e.g., gray, white, etc.) correctly. For example, an image of a piece ofwhite paper under an incandescent light may appear yellow whereas thehuman eye perceives the piece of paper as white. To account for thedifference between the color of the images that the sensor captures andwhat the human eye perceives, the WB module 435 adjusts the color valuesof the image so that the captured image properly reflects the colorsperceived by the human eye.

The statistics engine 465 collects image data at various stages of theCIPU processing pipeline 485. For example, FIG. 4 shows that thestatistics engine 465 collects image data after the LS module 425, thedemosaicing module 430, and the WB module 435. Different embodimentscollect data from any number of different stages of the CIPU processingpipeline 485. The statistics engine 465 processes the collected data,and, based on the processed data, adjusts the operations of the camerasensors 405 a and 405 b through the controller module 475 and the sensormodule 415. Examples of such operations include exposure and focus.Although FIG. 4 shows the statistics engine 465 controlling the camerasensors 405 a and 405 b through the controller module 475, otherembodiments of the statistics engine 465 control the camera sensorsthrough just the sensor module 415.

The processed data can also be used to adjust the operations of variousmodules of the CIPU 400. For instance, the statistics engine 465 of someembodiments adjusts the operations of the WB module 435 based on datacollected after the WB module 435. In some of such embodiments, thestatistics engine 465 provides an automatic white balance (AWB) functionby using the processed data to adjust the white balancing operation ofthe WB module 435. Other embodiments can use processed data collectedfrom any number of stages of the CIPU processing pipeline 485 to adjustthe operations of any number of modules within the CIPU processingpipeline 485. Further, the statistics engine 465 can also receiveinstructions from the controller module 475 to adjust the operations ofone or more modules of the CIPU processing pipeline 485.

After receiving the images from the WB module 435, the gamma module 440performs a gamma correction operation on the image to code and decodeluminance or tristimulus values of the camera system. The gamma module440 of some embodiments corrects gamma by converting a 10-12 bit linearsignal into an 8 bit non-linear encoding in order to correct the gammaof the image. Some embodiments correct gamma by using a lookup table.

The CSC module 445 converts the image received from the gamma module 440from one color space to another color space. Specifically, the CSCmodule 445 converts the image from an RGB color space to a luminance andchrominance (YUV) color space. However, other embodiments of the CSCmodule 445 can convert images from and to any number of color spaces.

The HSC module 450 may adjust the hue, saturation, contrast, or anycombination thereof of the images received from the CSC module 445. TheHSC module 450 may adjust these properties to reduce the noise orenhance the images, for example. For instance, the saturation of imagescaptured by a low-noise camera sensor can be increased to make theimages appear more vivid. In contrast, the saturation of images capturedby a high-noise camera sensor can be decreased to reduce the color noiseof such images.

After the HSC module 450, the scaler module 455 may resize images toadjust the pixel resolution of the image or to adjust the data size ofthe image. The scaler module 455 may also reduce the size of the imagein order to fit a smaller display, for example. The scaler module 455can scale the image a number of different ways. For example, the scalermodule 455 can scale images up (i.e., enlarge) and down (i.e., shrink).The scaler module 455 can also scale images proportionally or scaleimages anamorphically.

The filter module 460 applies one or more filter operations to imagesreceived from the scaler module 455 to change one or more attributes ofsome or all pixels of an image. Examples of filters include a low-passfilter, a high-pass filter, a band-pass filter, a bilateral filter, aGaussian filter, among other examples. As such, the filter module 460can apply any number of different filters to the images.

The controller module 475 of some embodiments is a microcontroller thatcontrols the operation of the CIPU 400. In some embodiments, thecontroller module 475 controls (1) the operation of the camera sensors(e.g., exposure level) through the sensor module 415, (2) the operationof the CIPU processing pipeline 485, (3) the timing of the CIPUprocessing pipeline 485 (e.g., when to switch camera sensors, when toswitch registers, etc.), and (4) a flash/strobe (not shown), which ispart of the dual camera mobile device of some embodiments.

Some embodiments of the controller module 475 process instructionsreceived from the statistics engine 465 and the CIPU driver 480. In someembodiments, the instructions received from the CIPU driver 480 areinstructions from the dual camera mobile device (i.e., received from thelocal device) while in other embodiments the instructions received fromthe CIPU driver 480 are instructions from another device (e.g., remotecontrol during a video conference). Based on the processed instructions,the controller module 475 can adjust the operation of the CIPU 400 byprogramming the values of the registers 470. Moreover, the controllermodule 475 can dynamically reprogram the values of the registers 470during the operation of the CIPU 400.

As shown in FIG. 4, the CIPU 400 includes a number of modules in theCIPU processing pipeline 485. However, one of ordinary skill willrealize that the CIPU 400 can be implemented with just a few of theillustrated modules or with additional and different modules. Inaddition, the processing performed by the different modules can beapplied to images in sequences different from the sequence illustratedin FIG. 4.

An example operation of the CIPU 400 will now be described by referenceto FIG. 4. For purposes of explanation, the set of registers Ra is usedfor processing images captured by camera sensor 405 a of the dual cameramobile device and the set of registers Rb is used for processing imagescaptured by camera sensor 405 b of the dual camera mobile device. Thecontroller module 475 receives instructions from the CIPU driver 480 toproduce images captured by one of the cameras of the dual camera mobiledevice.

The controller module 475 then initializes various modules of the CIPUprocessing pipeline 485 to process images captured by one of the camerasof the dual camera mobile device. In some embodiments, this includes thecontroller module 475 checking that the correct set of registers of theregisters 470 are used. For example, if the CIPU driver 480 instructsthe controller module 475 to produce images captured by the camerasensor 405 a, the controller module 475 checks that the set of registersRa is the set of registers from which the modules of the CIPU 400 read.If not, the controller module 475 switches between the sets of registersso that the set of registers Ra is the set that is read by the modulesof the CIPU 400.

For each module in the CIPU processing pipeline 485, the mode ofoperation is indicated by the values stored in the set of registers Ra.As previously mentioned, the values in the set of registers 470 can bedynamically reprogrammed during the operation of the CIPU 400. Thus, theprocessing of one image can differ from the processing of the nextimage. While the discussion of this example operation of the CIPU 400describes each module in the CIPU 400 reading values stored in registersto indicate the mode of operation of the modules, in somesoftware-implemented embodiments, parameters are instead passed to thevarious modules of the CIPU 400.

In some embodiments, the controller module 475 initializes the sensormodule 415 by instructing the sensor module 415 to delay a particularamount of time after retrieving an image from the pixel array 410 a. Inother words, the controller module 475 instructs the sensor module 415to retrieve the images from the pixel array 410 a at a particular rate.

Next, the controller module 475 instructs the camera sensor 405 athrough the sensor module 415 to capture images. In some embodiments,the controller module 475 also provides exposure and other cameraoperation parameters to the camera sensor 405 a. In other embodiments,the camera sensor 405 a uses default values for the camera sensoroperation parameters. Based on the parameters, the camera sensor 405 acaptures a raw image, which is stored in the pixel array 410 a. Thesensor module 415 retrieves the raw image from the pixel array 410 a andsends the image to the line/frame buffer 417 for storage before the CIPUprocessing pipeline 485 processing the image.

Under certain circumstances, images may be dropped by the line/framebuffer 417. When the camera sensors 405 a and/or 405 b are capturingimages at a high rate, the sensor module 415 may receive and storeimages in the line/frame buffer 417 faster than the BPC module 420 canretrieve the images from the line/frame buffer 417 (e.g., capturing highframe-rate video), and the line/frame buffer 417 will become full. Whenthis happens, the line/frame buffer 417 of some embodiments drops images(i.e., frames) based on a first in, first out basis. That is, when theline/frame buffer 417 drops an image, the line/frame buffer 417 dropsthe image that was received before all the other images in theline/frame buffer 417.

The processing of the image by the CIPU processing pipeline 485 startsby the BPC module 420 retrieving the image from the line/frame buffer417 to correct any bad pixels in the image. The BPC module 420 thensends the image to the LS module 425 to correct for any unevenillumination in the image. After the illumination of the image iscorrected, the LS module 425 sends the image to the demosaicing module430 where it processes the raw image to generate an RGB image from theraw image. Next, the WB module 435 receives the RGB image from thedemosaicing module 430 and adjusts the white balance of the RGB image.

As noted above, the statistics engine 465 may have collected some dataat various points of the CIPU processing pipeline 485. For example, thestatistics engine 465 collects data after the LS module 425, thedemosaicing module 430, and the WB module 435 as illustrated in FIG. 4.Based on the collected data, the statistics engine 465 may adjust theoperation of the camera sensor 405 a, the operation of one or moremodules in the CIPU processing pipeline 485, or both, in order to adjustthe capturing of subsequent images from the camera sensor 405 a. Forinstance, based on the collected data, the statistics engine 465 maydetermine that the exposure level of the current image is too low andthus instruct the camera sensor 405 a through the sensor module 415 toincrease the exposure level for subsequently captured images. Thus, thestatistics engine 465 of some embodiments operates as a feedback loopfor some processing operations.

After the WB module 435 adjusts the white balance of the image, it sendsthe image to the gamma module 440 for gamma correction (e.g., adjustingthe gamma curve of the image). The CSC module 445 receives thegamma-corrected image from the gamma module 440 and performs color spaceconversion. In this example, the CSC module 445 converts the RGB imageto a YUV image. In other words, the CSC module 445 converts an imagethat is represented in an RGB color space to an image that isrepresented in a YUV color space. The HSC module 450 receives the YUVimage from the CSC module 445 and adjusts the hue, saturation, andcontrast attributes of various pixels in the image. After the HSC module450, the scaler module 455 resizes the image (e.g., enlarging orshrinking the image). The filter module 460 applies one or more filterson the image after receiving the image from the scaler module 455.Finally, the filter module 460 sends the processed image to the CIPUdriver 480.

In this example of the operation of the CIPU 400 described above, eachmodule in the CIPU processing pipeline 485 processed the image in somemanner. However, other images processed by the CIPU 400 may not requireprocessing by all the modules of the CIPU processing pipeline 485. Forexample, an image may not require white balance adjustment, gammacorrection, scaling, or filtering. As such, the CIPU 400 can processimages any number of ways based on a variety of received input such asinstructions from the CIPU driver 480 or data collected by the statisticengine 465, for example.

Different embodiments control the rate at which images are processed(i.e., frame rate) differently. One manner of controlling the frame rateis through manipulation of vertical blanking intervals (VBI). For someembodiments that retrieve image lines for processing images on aline-by-line basis, a VBI is the time difference between retrieving thelast line of an image of a video captured by a camera of the dual cameramobile device from a pixel array and retrieving the first line of thenext image of the video from the pixel array. In other embodiments, aVBI is the time difference between retrieving one image of a videocaptured by a camera of the dual camera mobile device from a pixel arrayand retrieving the next image of the video the pixel array.

One example where VBI can be used is between the sensor module 415 andthe pixel arrays 410 a and 410 b. For example, some embodiments of thesensor module 415 retrieve images from the pixel arrays 410 a and 410 bon a line-by-line basis and other embodiments of the sensor module 415retrieve images from the pixel arrays 410 a and 410 b on animage-by-image basis. Thus, the frame rate can be controlled byadjusting the VBI of the sensor module 415: increasing the VBI reducesthe frame rate and decreasing the VBI increases the frame rate.

FIG. 5 conceptually illustrates examples of different frame rates 505,510, and 515 based on different VBIs. Each sequence shows an image,which is captured by one of the cameras of the dual camera mobiledevice, of a person holding a guitar at various time instances 525-555along timeline 520. In addition, the time between each time instance525-555 is the same and will be referred to as one time unit. Forpurposes of explanation, FIG. 5 will now be described by reference tothe sensor module 415 and the pixel array 410 a of FIG. 4. As such, eachimage represents a time instance along the timeline 520 at which thesensor module 415 retrieves an image from the pixel array 410 a.

In the example frame rate 505, the VBI of the sensor module 415 for thepixel array 410 a is set to three time units (e.g., by the controllermodule 475). That is, the sensor module 415 retrieves an image from thepixel array 410 a every third time instance along the timeline 520. Asshown in the example frame rate 505, the sensor module 415 retrieves animage at the time instances 525, 540, and 555. Thus, the example framerate 505 has a frame rate of one image per three time units.

The example frame rate 510 is similar to the example frame rate 505except the VBI is set to two time units. Thus, the sensor module 415retrieves an image from the pixel array 410 a every second time instancealong the timeline 520. The example frame rate 510 shows the sensormodule 415 retrieving an image from the pixel array 410 a at the timeinstances 525, 535, 545, and 555. Since the VBI of the example framerate 510 is less than the VBI of the example frame rate 505, the framerate of the example frame rate 510 is higher than the frame rate of theexample frame rate 505.

The example frame rate 515 is also similar to the example frame rate 505except the VBI of the sensor module 415 for the pixel array 410 a is setto one time unit. Therefore, the sensor module 415 is instructed toretrieve an image from the pixel array 410 a every time instance alongthe timeline 520. As illustrated, the sensor module 415 retrieves animage from the pixel array 410 a at the time instances 525-555. The VBIof the example frame rate 515 is less than the VBIs of the example framerates 505 and 510. Therefore, the frame rate of the example frame rate515 is higher than the example frame rates 505 and 510.

III. Video Conferencing

A. Video Conference Architecture

FIG. 6 conceptually illustrates a software architecture for a videoconferencing and processing module 600 of a dual camera mobile device ofsome embodiments. The video conferencing and processing module 600includes a CIPU driver 605, a media exchange module 610, and an encoderdriver 620 that are similar to the corresponding modules and drivers305, 310, and 320 described above by reference to FIG. 3. The videoconferencing and processing module 600 also includes a video conferencemodule 625, a video conference client 645, and a network interface 650for performing a variety of video conferencing functions. Like the videoprocessing and encoding module 300, the video conferencing andprocessing module 600 processes and encodes images that are capturedfrom cameras of the dual camera mobile device.

As described above by reference to FIG. 3, the media exchange module 610allows consumers and producers of media content in the device toexchange media content and instructions regarding the processing of themedia content, the CIPU driver 605 serves as a communication interfacewith the captured image processing unit (CIPU) 655, and the encoderdriver 620 serves as a communication interface with the encoder hardware660 (e.g., an encoder chip, an encoding component on a system on chip,etc.).

The video conference module 625 of some embodiments handles variousvideo conferencing functions such as image processing, video conferencemanagement, and networking. As shown, the video conference module 625interacts with the media exchange module 610, the video conferenceclient 645, and the network interface 650. In some embodiments, thevideo conference module 625 receives instructions from and sendsinstructions to the video conference client 645. The video conferencemodule 625 of some embodiments also sends data to and receives data fromnetworks (e.g., a local area network (LAN), a wireless local areanetwork (WLAN), a wide area network (WAN), a network of networks, a codedivision multiple access (CDMA) network, a GSM network, etc.) throughthe network interface 650.

The video conference module 625 includes an image processing layer 630,a management layer 635, and a network layer 640. In some embodiments,the image processing layer 630 performs image processing operations onimages for video conferencing. For example, the image processing layer630 of some embodiments performs exposure adjustment, image resizing,perspective correction, and dynamic range adjustment as described infurther detail below. The image processing layer 630 of some embodimentssends requests through the media exchange module 610 for images from theCIPU 655.

The management layer 635 of some embodiments controls the operation ofthe video conference module 625. For instance, in some embodiments, themanagement layer 635 initializes a camera/cameras of the dual cameramobile device, processes images and audio to transmit to a remotedevice, and processes images and audio received from the remote device.In some embodiments, the management layer 635 generates composite (e.g.,PIP) displays for the device. Moreover, the management layer 635 maychange the operation of the video conference module 625 based onnetworking reports received from the network layer 640.

In some embodiments, the network layer 640 performs some or all of thenetworking functionalities for video conferencing. For instance, thenetwork layer 640 of some embodiments establishes a network connection(not shown) between the dual camera mobile device and a remote device ofa video conference, transmits images to the remote device, and receivesimages from the remote device, among other functionalities, as describedbelow and in the above-incorporated U.S. patent application Ser. No.______, entitled “Establishing Video Conference During a Phone Call,”with Attorney Docket Number APLE.P0212. In addition, the network layer640 receives networking data such as packet loss, one-way latency, androundtrip delay time, among other types of data, processes such data,and reports the data to the management layer 635.

The video conference client 645 of some embodiments is an applicationthat may use the video conferencing functions of the video conferencemodule 625 such as a video conferencing application, a voice-over-IP(VOIP) application (e.g., Skype), or an instant messaging application.In some embodiments, the video conference client 645 is a stand-aloneapplication while in other embodiments the video conference client 645is integrated into another application.

In some embodiments, the network interface 650 is a communicationinterface that allows the video conference module 625 and the videoconference client 645 to send data and receive data over a network(e.g., a cellular network, a local area network, a wireless network, anetwork of networks, the Internet, etc.) through the network interface650. For instance, if the video conference module 625 wants to send data(e.g., images captured by cameras of the dual camera mobile device) toanother device on the Internet, the video conference module 625 sendsthe images to the other device through the network interface 650.

B. Video Conference Set Up

FIG. 7 conceptually illustrates an example video conference requestmessaging sequence 700 of some embodiments. This figure shows the videoconference request messaging sequence 700 among a video conferenceclient 710 running on a device 705, a video conference server 715, and avideo conference client 725 running on a device 720. In someembodiments, the video conference clients 710 and 725 are the same asthe video conference client 645 shown in FIG. 6. As shown in FIG. 7, onedevice (i.e., the device 705) requests a video conference and anotherdevice (i.e., the device 720) responds to such request. The dual cameramobile device described in the present application can perform bothoperations (i.e., make a request and respond to a request).

The video conference server 715 of some embodiments routes messagesamong video conference clients. While some embodiments implement thevideo conference server 715 on one computing device, other embodimentsimplement the video conference server 715 on multiple computing devices.In some embodiments, the video conference server is a publiclyaccessible server that can handle and route messages for numerousconferences at once. Each of the video conference clients 710 and 725 ofsome embodiments communicates with the video conference server 715 overa network (e.g., a cellular network, a local area network, a wirelessnetwork, a network of networks, the Internet etc.) through a networkinterface such as the network interface 650 described above.

The video conference request messaging sequence 700 of some embodimentsstarts when the video conference client 710 receives (at operation 1) arequest from a user of the device 705 to start a video conference withthe device 720. The video conference client 710 of some embodimentsreceives the request to start the video conference when the user of thedevice 705 selects a user interface (UI) item of a user interfacedisplayed on the device 705. Examples of such user interfaces areillustrated in FIG. 8 and FIG. 11, which are described below.

After the video conference client 710 receives the request, the videoconference client 710 sends (at operation 2) a video conference request,which indicates the device 720 as the recipient based on input from theuser, to the video conference server 715. The video conference server715 forwards (at operation 3) the video conference request to the videoconference client 725 of the device 720. In some embodiments, the videoconference server 715 forwards the video conference request to the videoconference client 725 using push technology. That is, the videoconference server 715 initiates the transmission of the video conferencerequest to the video conference client 725 upon receipt from the videoconference client 710, rather than waiting for the client 725 to send arequest for any messages.

When the video conference client 725 of some embodiments receives thevideo conference request, a user interface is displayed on the device720 to indicate to the user of the device 720 that the user of thedevice 705 sent a request to start a video conference and to prompt theuser of the device 720 to accept or reject the video conference request.An example of such a user interface is illustrated in FIG. 9, which isdescribed below. In some embodiments, when the video conference client725 receives (at operation 4) a request to accept the video conferencerequest from the user of the device 705, the video conference client 725sends (at operation 5) a video conference acceptance to the videoconference server 715. The video conference client 725 of someembodiments receives the request to accept the video conference requestwhen the user of the device 720 selects a user interface item of a userinterface as illustrated in FIG. 9, for example.

After the video conference server 715 receives the video conferenceacceptance from the video conference client 725, the video conferenceserver 715 forwards (at operation 6) the video conference acceptance tothe video conference client 710. Some embodiments of the videoconference server 715 forward the video conference acceptance to thevideo conference client 710 using the push technology described above.

Upon receiving the video conference acceptance, some embodimentsestablish (at operation 7) a video conference between the device 705 andthe device 720. Different embodiments establish the video conferencedifferently. For example, the video conference establishment of someembodiments includes negotiating a connection between the device 705 andthe device 720, determining a bit rate at which to encode video, andexchanging video between the device 705 and the device 720.

In the above example, the user of the device 720 accepts the videoconference request. In some embodiments, the device 720 can beconfigured (e.g., through the preference settings of the device) toautomatically accept incoming video conference requests withoutdisplaying a UI. Moreover, the user of the device 720 can also reject(at operation 4) the video conference request (e.g., by selecting a userinterface item of a user interface displayed on the device 720). Insteadof sending a video conference acceptance, the video conference client725 sends a video conference rejection to the video conference server715, which forwards the video conference rejection to the videoconference client 710. The video conference is then never established.

In some embodiments, a video conference is initiated based on an ongoingphone call. That is, while the user of a mobile device is engaged in aphone call with a second user, the user can turn the phone call into avideo conference with the permission of the other party. For someembodiments of the invention, FIG. 8 illustrates the start of such avideo conference by a dual camera handheld mobile device 800. Thisfigure illustrates the start of the video conference in terms of fiveoperational stages 810, 815, 820, 825, and 830 of a user interface(“UI”) 805 of the device 800.

As shown in FIG. 8, the UI 805 includes a name field 835, a selectionmenu 840, and a selectable UI item 845. The name field 835 displays thename of the person on the other end of the phone call, with whom a userwould like to request a video conference. In this example, theselectable UI item 845 (which can be implemented as a selectable button)provides a selectable End Call option for the user to end the phonecall. The selection menu 840 displays a menu of selectable UI items,such as a Speakerphone item 842, a Mute item 844, a Keypad item 846, aPhonebook item 848, a Hold item 852, a Video Conference item 854, etc.Different embodiments display the selection menu differently. For theembodiments illustrated by FIG. 8, the selection menu 840 includesseveral equally sized icons, each of which represents a differentoperation. Other embodiments provide a scrollable menu, or give priorityto particular items (e.g., by making the items larger).

The operation of the UI 805 will now be described by reference to thestate of this UI during the five stages, 810, 815, 820, 825, and 830that are illustrated in FIG. 8. In the first stage 810, a phone call hasbeen established between the handheld mobile device user and NancyJones. The second stage 815 displays the UI 805 after the user selectsthe selectable Video Conference option 854 (e.g., through a singlefinger tap by finger 850) to activate a video conference tool. In thisexample, the Video Conference option 854 (which can be implemented as aselectable icon) allows the user to start a video conference during thephone call. In the second stage, the Video Conference option 854 ishighlighted to indicate that the video conference tool has beenactivated. Different embodiments may indicate such a selection indifferent ways (e.g., by highlighting the border or the text of theitem).

The third stage 820 displays the UI 805 after the device 800 has startedthe video conference process with the selection of the Video Conferenceoption 854. The third stage is a transitional hold stage while thedevice waits for the video conference to be established (e.g., while thedevice waits for the device on the other end of the call to accept orreject the video conference). In the third stage 820, the user of thedevice 800 can still talk to the user of the other device (i.e., NancyJones) while the video conference connection is being established. Inaddition, some embodiments allow the user of the device 800 to cancelthe video conference request in the third stage 820 by selecting aselectable UI item displayed on the UI 805 (not shown) for canceling thevideo conference request. During this hold stage, different embodimentsuse different displays in the UI 805 to indicate the wait state.

As shown in FIG. 8, in some embodiments the wait state of the thirdstage is illustrated in terms of a full screen display of a video beingcaptured by the device 800 along with a “Preview” notation at the bottomof this video. Specifically, in FIG. 8, the third stage 820 illustratesthe start of the video conference process by displaying in a displayarea 860 of the UI 805 a full screen presentation of the video beingcaptured by the device's camera. In some embodiments, the front camerais the default camera selected by the device at the start of a videoconference. Often, this front camera points to the user of the device atthe start of the video conference. Accordingly, in the exampleillustrated in FIG. 8, the third stage 820 illustrates the device 800 aspresenting a full screen video of the user of the device 800. The waitstate of the device is further highlighted by the “Preview” designation865 below the video appearing in the display area 860 during the thirdstage 820.

The transitional third hold stage 820 can be represented differently insome embodiments. For instance, some embodiments allow the user of thedevice 800 to select the back camera as the camera for starting thevideo conference. To allow for this selection, some embodiments allowthe user to specify (e.g., through a menu preference setting) the backcamera as the default camera for the start of a video conference, and/orallow the user to select the back camera from a menu that displays theback and front cameras after the user selects the Video Conferenceoption 854. In either of these situations, the UI 805 (e.g., displayarea 860) displays a video captured by the back camera during the thirdhold stage 820.

Also, other embodiments might indicate the activation of the videoconference tool by displaying the smaller version of the video capturedby the device 800, by displaying a still image that is stored on thedevice 800, by providing a message to highlight the wait state of thedevice (e.g., by showing “Conference Being Established”), by notdisplaying the “Preview” designation, etc. Also, in the third stage 820,the UI 805 of some embodiments provides an End button (not shown) toallow the user to cancel entering the video conference and revert backto the phone call if he decides not to enter the video conference atthis stage (e.g., while the user is waiting for the remote user torespond to his request).

The fourth stage 825 illustrates the UI 805 in a transitional stateafter the remote user has accepted the video conference request and avideo conference connection has been established. In this transitionalstate, the display area 860 that displays the video of the local user(that is being captured by the front camera in this example) graduallydecreases in size (i.e., gradually shrinks), as indicated by the arrows875. The display area 860 (i.e., the local user's video) shrinks so thatthe UI 805 can display a display area 870 (e.g., a display window 870)that contains the video from a camera of the remote device behind thedisplay area 860. In other words, the shrinking of the local user'svideo 860 creates a PIP display 880 that has a foreground inset display860 of the local user's video and a background main display 870 of theremote user. In this example, the background main display 870 presents avideo of a lady whose video is being captured by the remote device'sfront camera (e.g., Nancy Jones, the user of the remote device) or alady whose video is being captured by the remote device's back camera(e.g., a lady whose video is being captured by Nancy Jones). One ofordinary skill will realize that the transitional fourth stage shown inFIG. 8 is simply one exemplary approach used by some embodiments, andthat other embodiments might animate the transitional fourth stagedifferently.

The fourth stage 825 also illustrates a selectable UI item 832 in alower display area 855. The selectable UI item 832 (which can beimplemented as a selectable button) provides a selectable End Conferenceoption 832 below the PIP display 880. The user may select this EndConference option 832 to end the video conference (e.g., through asingle finger tap). Different embodiments may allow the user to end theconference in different ways, such as by toggling a switch on the mobiledevice, by giving voice commands, etc. Moreover, different embodimentsmay allow the End Conference option 832 to fade away during the videoconference, thereby allowing the PIP display 880) to take up the entiredisplay area 885. The End Conference option 832 may then reappear at asingle finger tap at the bottom of the display area 885, giving the useraccess to the End Conference option 832. In some embodiments, the layoutof the display area 855 is same as the display area 855 described infurther detail below.

The fifth stage 830 illustrates the UI 805 after the animation of thefourth transitional state 825 has ended. Specifically, the fifth stage830 illustrates a PIP display 880 that is presented by the UI 805 duringthe video conference. As mentioned above, this PIP display 880 includestwo video displays: a larger background display 870 from the remotecamera and a smaller foreground inset display 860 from the local camera.

This PIP display 880 is only one manner of presenting a composite viewof the videos being captured by the remote and local devices. Inaddition to this composite view, the devices of some embodiments provideother composite views. For example, instead of having a largerbackground display 870 of the remote user, the larger background display870 can be of the local user and the smaller foreground inset display860 of the remote user. As further described below, some embodimentsallow a user to switch during a video conference between the localcameras and/or remote cameras as the cameras for the inset and mainviews in the PIP display 880.

Also, some embodiments allow the local and remote videos to appear inthe UI 805 in two side-by-side display areas (e.g., left and rightdisplay windows, or top and bottom display windows) or two diagonallyaligned display areas. The manner of the PIP display or a defaultdisplay mode may be specified by the user in some embodiments throughthe preference settings of the device or through controls that the usercan select during a video conference, as further described below and inthe above-incorporated U.S. patent application Ser. No. ______, entitled“Establishing Video Conference During a Phone Call,” with AttorneyDocket Number APLE.P0212.

When the user of the device 800 of FIG. 8 invites the remote user to avideo conference, the remote user may accept or reject the invitation.FIG. 9 illustrates a UI 905 of the remote user's device 900 at sixdifferent stages 910, 915, 920, 925, 930, and 935 that show the sequenceof operations for presenting and accepting a video conference invitationat the remote user's device. The description of the UI 905 below refersto the user of the device 900 (i.e., the device that receives the videoconference request) as the invite recipient, and the user of the device800 (i.e., the device that sends the video conference request) as theinvite requestor. Also, in this example, it is assumed that the inviterecipient's device 900 is a dual camera device, like that of the inviterequestor. However, in other examples, one or both of these devices aresingle camera devices.

The first stage 910 illustrates the UI 905 when the invite recipientreceives an invitation to a video conference from the invite requestor,John Smith. As shown in FIG. 9, the UI 905 in this stage includes a namefield 995, a message field 940, and two selectable UI items 945 and 950.The name field 995 displays the name of a person who is requesting avideo conference. In some embodiments, the name field 995 displays aphone number of the person who is requesting a video conference insteadof the name of the person. The message field 940 displays an invite fromthe invite requestor to the invite recipient. In this example, the“Video Conference Invitation” in the field 940 indicates that the inviterequestor is requesting a video conference with the invite recipient.The selectable UI items 945 and 950 (which can be implemented asselectable buttons) provide selectable Deny Request and Accept Requestoptions 945 and 950 for the invite recipient to use to reject or acceptthe invitation. Different embodiments may display these optionsdifferently and/or display other options.

Upon seeing the “Video Conference Invitation” notation displayed in themessage field 940, the invite recipient may deny or accept the requestby selecting the Deny Request option 945 or Accept Request option 950 inthe UI, respectively. The second stage 915 illustrates that in theexample shown in FIG. 9, the user selects the Accept Request option 950.In this example, this selection is made by the user's finger tapping onthe Accept Request option 950, and this selection is indicated throughthe highlighting of this option 950. Other techniques are provided insome embodiments to select the Accept or Deny Request options 945 and950 (e.g., double-tapping, etc.) to indicate the selection (e.g.,highlighting the border or text of the UI item).

The third stage 920 displays the UI 905 after the invite recipient hasagreed to join the video conference. In this stage, the UI 905 entersinto a preview mode that shows a full screen presentation of the videofrom the remote device's front camera in a display area 944. The frontcamera in this case is pointed to the user of the remote device (i.e.,Nancy Jones in this example). Accordingly, her image is shown in thispreview mode. This preview mode allows the invite recipient to make surethat her video is displayed properly and that she is happy with herappearance before the video conference begins (e.g., before actualtransmission of the video begins). In some embodiments, a notation, suchas a “Preview” notation, may be displayed below the display area 944 toindicate that the invite recipient is in the preview mode.

Some embodiments allow the invite recipient to select the back camera asthe default camera for the start of the video conference, or to selectthe front or back camera at the beginning of the video conference, asfurther described in the above-incorporated U.S. patent application Ser.No. ______, entitled “Establishing Video Conference During a PhoneCall,” with Attorney Docket Number APLE.P0212. Also, other embodimentsdisplay the preview display of the invite recipient differently (e.g.,in a smaller image placed in the corner of the display area 944). Yetother embodiments do not include this preview mode, but rather start thevideo conference immediately after the invite recipient accepts therequest.

In the third stage, the UI 905 shows two selectable UI items 975 and946, one of which overlaps the display area 944 while the other is belowthis display area 944. The selectable UI item 975 is an Accept button975 that the user may select to start video conferencing. The selectableUI item 946 is an End button 946 that the invite recipient can select ifshe decides not to join the video conference at this stage.

The fourth stage 925 displays the UI 905 after the invite recipientselects the Accept button 975. In this example, the Accept button 975 ishighlighted to indicate that the invite recipient is ready to start thevideo conference. Such a selection may be indicated in different ways inother embodiments.

The fifth stage 930 illustrates the UI 905 in a transitional state afterthe invite recipient has accepted the video conference request. In thistransitional stage, the display area 944 that displays the video of theinvite recipient (that is being captured by the front camera in thisexample) gradually decreases in size (i.e., gradually shrinks), asindicated by the arrows 960. The invite recipient's video shrinks sothat the UI 905 can display a display area 965 (e.g., a display window965) that contains the video from a camera of the invite requestorbehind the display area 944. In other words, the shrinking of the inviterecipient's video creates a PIP display 980 that has a foreground insetdisplay area 944 of the invite recipient's video and a background maindisplay 965 of the invite requestor.

In this example, the background main display 965 presents a video of aman whose video is being captured by the local device's front camera(i.e., John Smith, the user of the local device 800). In anotherexample, this video could have been that of a man whose video is beingcaptured by the local device's back camera (e.g., a man whose video isbeing captured by John Smith). Different embodiments may animate thistransitional fifth stage differently.

The UI at the fifth stage 930 also displays a display area 855 (e.g., atool bar or a menu bar) that includes selectable UI item 985 (e.g., mutebutton 985) for muting the audio of the other user during the videoconference, selectable UI item 987 (e.g., end conference button 987) forending the video conference, and selectable UI item 989 (e.g., switchcamera button 989) for switching cameras, which is described in furtherdetail below. As such, the invite recipient may select any of theselectable UI items 985-989 (e.g., through a single finger tap) toperform the desired operation during the video conference. Differentembodiments may allow the invite recipient to perform any of theoperations in different ways, e.g., by toggling a switch on the mobiledevice, by giving voice commands, etc.

Although FIG. 9 shows an example layout for the display area 855, someembodiments provide different layouts of the display area 855 such asthe layout of display area 855 of FIG. 8, which includes just aselectable End Conference UI item 832 for ending the video conference.Other layouts of display area 855 can include any number of differentselectable UI items for performing different functions. Moreover, thefifth stage 930 shows the display area 855 displayed at the bottom ofthe UI 905. Different embodiments of the display area 855 can bedisplayed at different locations within the UI 905 and/or defined asdifferent shapes.

FIG. 9 shows the display area 855 as a static display area (i.e., thedisplay area 855 is always displayed). However, in some embodiments thedisplay area 855 is a dynamic display area. In some such embodiments,the display area 855 is not ordinarily displayed. Rather, the displayarea 855 is displayed when a triggering event is received (e.g., a userselection such tapping the display area 980 once, a voice command,etc.). The display area 855 disappears after a user selection isreceived (e.g., selecting the selectable mute UI item 985) or a definedamount of time (e.g., 3 seconds), which can be specified by the userthrough the preference settings of the mobile device or the videoconference application. In some such embodiments, the display area 855is automatically displayed after the video conference starts anddisappears in the same manner mentioned above.

The sixth stage 935 illustrates the UI 905 after the animation of thefifth transitional stage has ended. Specifically, the sixth stageillustrates a PIP display 980 that is presented by the UI 905 during thevideo conference. As mentioned above, this PIP display 980 includes twovideo displays: a larger background display 965 from the local cameraand a smaller foreground inset display 944 from the remote camera. ThisPIP display 980 is only one manner of presenting a composite view of thevideos being captured by the remote and local devices. In addition tothis composite view, the devices of some embodiments provide othercomposite views. For example, instead of having a larger backgrounddisplay of the invite recipient, the larger background display can be ofthe invite requestor and the smaller foreground inset display of theinvite recipient. As further described in the above-incorporated U.S.patent application Ser. No. ______, entitled “Establishing VideoConference During a Phone Call,” with Attorney Docket Number APLE.P0212,some embodiments allow a user to control the inset and main views in aPIP display to switchably display the local and remote cameras. Also,some embodiments allow the local and remote videos to appear in the UI905 in two side-by-side display areas (e.g., left and right displaywindows, or top and bottom display windows) or two diagonally aligneddisplay areas. The manner of PIP display or a default display mode maybe specified by the user in some embodiments through the preferencesettings of the device or through controls that the user can selectduring a video conference, as further described in theabove-incorporated U.S. patent application Ser. No. ______, entitled“Establishing Video Conference During a Phone Call,” with AttorneyDocket Number APLE.P0212.

Although FIG. 9 shows the sequence of operations for presenting andaccepting a video conference invitation in terms of six differentoperational stages, some embodiments may implement the operation in lessstages. For instance, some of such embodiments may omit presenting thethird and fourth stages 920 and 925 and go from the second stage 915 tothe fifth stage 930 after the user selects the Accept Request option950. Other embodiments that implement that operation (i.e., presentingand accepting a video conference invitation) in less stages may omit thefirst and second stages 910 and 915 and present the user with the thirdstage 920 when the invite recipient receives an invitation to a videoconference from the invite requestor.

FIG. 10 illustrates an example of performing the operation illustratedin FIG. 9 in less stages by combining the first and third stages intoone stage and the second and fourth stage into one stage. In particular,this figure illustrates a UI 905 of the remote user's device 900 at fivedifferent stages 1090, 1092, 1094, 930, and 935. The first stage 1090 issimilar to the stage 810 except the name field 995 displays the name“John Smith” to indicate the name of the person on the other end of thetelephone call. That is, a phone call has been established between theuser of the remote mobile device and the user of the local device (i.e.,John Smith in this example). The second and third stages 1092 and 1094are similar to the first and second stages 910 and 915 of FIG. 9 exceptthe second and third stage 1092 and 1094 also show a preview of the userof the remote mobile device (i.e., Nancy Jones in this example). Thefourth and fifth stages 930 and 935 are the same as the fifth and sixthstages 930 and 935 of FIG. 9.

In addition to activating the video conference tool through a selectableoption during a phone call, some embodiments allow a user of a dualcamera device to initiate a video conference directly without having tomake a phone call first. FIG. 11 illustrates another such alternativemethod to initiate a video conference. This figure illustrates the UI1105 at seven different stages 1110, 1115, 1120, 1125, 1130, 1135, and1140 that show an alternative sequence of operations for starting avideo conference.

In the first stage 1110, a user is looking through a contacts list onthis mobile device for the person with whom he wants to engage in avideo conference, similar to how he would find a contact to call. In thesecond stage 1115, the user selects the person 1155 with whom he wouldlike to have a video conference (e.g., through a single finger tap 1160on the person's name 1155). This selection triggers the UI 1105 todisplay the contact's information and various user selectable options.In this example, Jason's name 1155 is highlighted to indicate that thisis the person with whom the user would like to have a video conference.Different embodiments may indicate such a selection in different ways.While the second stage 1115 allows the user of the device 1100 to selecta person with whom the user would like to have a video conferencethrough a contact list, some embodiments allow the user to select theperson through a “Recents” call history that lists a particular numberor name of a person with whom the user of the device 1100 recently had avideo conference or a phone call.

In the third stage 1120, the UI 1105 displays the selected person'sinformation 1162 and various selectable UI items 1168, 1172, and 1170after the person's name 1155 has been selected. In this example, one ofthe various selectable UI items 1172 (which can be implemented as aselectable icon or button) provides a video conference tool. The VideoConference option 1172 allows the user to invite the person identifiedby the contact 1166 to a video conference. Different embodiments displaythe information 1162 and selectable UI items 1168, 1172, and 1170differently (e.g., in a different arrangement).

The fourth stage 1125 shows the user selecting the Video Conferenceoption 1172 (e.g., through a single finger tap). In this example, theVideo Conference option 1172 is highlighted to indicate that the videoconference tool 1172 has been activated. Such selections may beindicated differently in different embodiments (e.g., by highlightingthe text or border of the selected icon).

The fifth, sixth and seventh stages 1130, 1135, and 1140 are similar tothe third, fourth and fifth stages 820, 825, and 830 illustrated in FIG.8 and may be understood by reference to the discussion of those stages.In brief, the fifth stage 1130 illustrates a transitional holding stagethat waits for the remote user to respond to the invitation to a videoconference. The sixth stage 1135 illustrates that after the remote userhas accepted the video conference request, the display area 1180 (thatdisplays the video of the local user) gradually decreases in size so theUI 1105 can show a display area 1192 that contains the video from acamera of the remote user behind the display area 1180. In the seventhstage 1140, the PIP display 1147 is presented by the UI 1105 during thevideo conference. In some embodiments, the layout of display area 855 inthe sixth stage 1135 and the seventh stage 1140 is like the layout ofthe display area 855 of FIG. 9, described above.

FIGS. 7, 8, 9, 10, and 11 show several ways of establishing a videoconference. In some embodiments, during a telephone call, audio data(e.g., voice) is transmitted through one communication channel (over acommunication network like a circuit-switched communication network or apacket-switched communication network) and, during a video conference,audio data is transmitted through another communication channel. Thus,in such embodiments, audio data (e.g., voice) is transmitted through acommunication channel before the video conference is established, andonce the video conference is established, audio is transmitted through adifferent communication channel (instead of the communication channelused during the telephone call).

In order to provide a seamless transition (e.g., handoff) of audio datafrom the telephone call to the video conference, some embodiments do notterminate the telephone call before establishing the video conference.For instance, some embodiments establish a peer-to-peer video conferenceconnection (e.g., after completing the message sequence illustrated inFIG. 7) before terminating the phone call and starting to transmitaudio/video data through the peer-to-peer communication session.Alternatively, other embodiments establish a peer-to-peer videoconference connection (e.g., after completing the message sequenceillustrated in FIG. 7) and start transmitting audio/video data throughthe peer-to-peer communication session, before terminating the phonecall and starting to present the received audio/video data.

A peer-to-peer video conference connection of some embodiments allowsthe mobile devices in the video conference to directly communicate witheach other (instead of communicating through a central server, forexample). Some embodiments of a peer-to-peer video conference allow themobile devices in the video conferences to share resources with eachother. For instance, through a control communication channel of a videoconference, one mobile device can remotely control operations of anothermobile device in the video conference by sending instructions from theone mobile device to the other mobile device to direct the other mobiledevice to process images differently (i.e., share its image processingresource) such as an exposure adjustment operation, a focus adjustmentoperation, and/or a switch camera operation, described in further detailbelow.

C. Video Conference Architecture

As mentioned above, FIG. 12 conceptually illustrates a softwarearchitecture for a video conferencing and processing module 1200 of adual camera mobile device of some embodiments. As shown, the videoconferencing and processing module 1200 includes a client application1265, a video conference module 1202, a media exchange module 1220, abuffer 1225, a captured image processing unit (CIPU) driver 1230, anencoder driver 1235, and a decoder driver 1240. In some embodiments, thebuffer 1225 is a frame buffer that stores images of a video for displayon a display 1245 of the dual camera mobile device.

In some embodiments, the client application 1265 is the same as thevideo conference client 645 of FIG. 6. As mentioned above, the clientapplication 1265 may be integrated into another application orimplemented as a stand-alone application. The client application 1265may be an application that uses the video conferencing functions of thevideo conference module 1202, such as a video conferencing application,a voice-over-IP (VOIP) application (e.g., Skype), or an instantmessaging application.

The client application 1265 of some embodiments sends instructions tothe video conference module 1202 such as instructions to start aconference and end a conference, receives instructions from the videoconference module 1202, routes instructions from a user of the dualcamera mobile device to the video conference module 1202, and generatesuser interfaces that are displayed on the dual camera mobile device andallow a user to interact with the application.

D. Video Conference Manager

As shown in FIG. 12, the video conference module 1202 includes a videoconference manager 1204, an image processing manager 1208, a networkingmanager 1214, and buffers 1206, 1210, 1212, 1216, and 1218. In someembodiments, the video conference module 1202 is the same as the videoconference module 625 illustrated in FIG. 6 and thus performs some orall of the same functions described above for the video conferencemodule 625.

In some embodiments, the video conference manager 1204 is responsiblefor initializing some or all of the other modules of the videoconference module 1202 (e.g., the image processing manager 1208 and thenetworking manager 1214) when a video conference is starting,controlling the operation of the video conference module 1202 during thevideo conference, and ceasing the operation of some or all of the othermodules of the video conference module 1202 when the video conference isending.

The video conference manager 1204 of some embodiments also processesimages received from one or more devices in the video conference andimages captured by one of both cameras of the dual camera mobile devicefor display on the dual camera mobile device. For instance, the videoconference manager 1204 of some embodiments retrieves decoded images,that were received from another device participating in the videoconference, from the buffer 1218 and retrieves images processed by CIPU1250 (i.e., images captured by the dual camera mobile device) from thebuffer 1206. In some embodiments, the video conference manager 1204 alsoscales and composites the images before displaying the images on thedual camera mobile device. That is, the video conference manager 1204generates the PIP or other composite views to display on the mobiledevice in some embodiments. Some embodiments scale the images retrievedfrom the buffers 1206 and 1218 while other embodiments just scale imagesretrieved from one of the buffers 1206 and 1218.

Although FIG. 12 illustrates the video conference manager 1204 as partof the video conference module 1202, some embodiments of the videoconference manager 1204 are implemented as a component separate from thevideo conference module 1202. As such, a single video conference manager1204 can be used to manage and control several video conference modules1202. For instance, some embodiments will run a separate videoconference module on the local device to interact with each party in amulti-party conference, and each of these video conference modules onthe local device are managed and controlled by the one video conferencemanager.

The image processing manager 1208 of some embodiments processes imagescaptured by the cameras of the dual camera mobile device before theimages are encoded by the encoder 1255. For example, some embodiments ofthe image processing manager 1208 perform one or more of exposureadjustment, focus adjustment, perspective correction, dynamic rangeadjustment, and image resizing on images processed by the CIPU 1250. Insome embodiments, the image processing manager 1208 controls the framerate of encoded images that are transmitted to the other device in thevideo conference.

Some embodiments of the networking manager 1214 manage one or moreconnections between the dual camera mobile device and the other deviceparticipating in the video conference. For example, the networkingmanager 1214 of some embodiments establishes the connections between thedual camera mobile device and the other device of the video conferenceat the start of the video conference and tears down these connections atthe end of the video conference.

During the video conference, the networking manager 1214 transmitsimages encoded by the encoder 1255 to the other device of the videoconference and routes images received from the other device of the videoconference to decoder 1260 for decoding. In some embodiments, thenetworking manager 1214, rather than the image processing manager 1208,controls the frame rate of the images that are transmitted to the otherdevice of the video conference. For example, some such embodiments ofthe networking manager 1214 control the frame rate by dropping (i.e.,not transmitting) some of the encoded frames that are supposed to betransmitted to the other device of the video conference.

As shown, the media exchange module 1220 of some embodiments includes acamera source module 1222, a video compressor module 1224, and a videodecompressor module 1226. The media exchange module 1220 is the same asthe media exchange module 310 shown in FIG. 3, with more detailprovided. The camera source module 1222 routes messages and mediacontent between the video conference module 1202 and the CIPU 1250through the CIPU driver 1230, the video compressor module 1224 routesmessage and media content between the video conference module 1202 andthe encoder 1255 through the encoder driver 1235, and the videodecompressor module 1226 routes messages and media content between thevideo conference module 1202 and the decoder 1260 through the decoderdriver 1240. Some embodiments implement the TNR module 315 included inthe media exchange module 310 (not shown in FIG. 12) as part of thecamera source module 1222 while other embodiments implement the TNRmodule 315 as part of the video compressor module 1224.

In some embodiments, the CIPU driver 1230 and the encoder driver 1235are the same as the CIPU driver 305 and the encoder driver 320illustrated in FIG. 3. The decoder driver 1240 of some embodiments actsas a communication interface between the video decompressor module 1226and decoder 1260. In such embodiments, the decoder 1260 decodes imagesreceived from the other device of the video conference through thenetworking manager 1214 and routed through the video decompressor module1226. After the images are decoded, they are sent back to the videoconference module 1202 through the decoder driver 1240 and the videodecompressor module 1226.

In addition to performing video processing during a video conference,the video conferencing and processing module 1200 for the dual cameramobile device of some embodiments also performs audio processingoperations during the video conference. FIG. 13 illustrates such asoftware architecture. As shown, the video conferencing and processingmodule 1200 includes the video conference module 1202 (which includesthe video conference manager 1204, the image processing manager 1208,and the networking manager 1214), the media exchange module 1220, andthe client application 1265. Other components and modules of the videoconferencing and processing module 1200 shown in FIG. 12 are omitted inFIG. 13 to simplify the description. The video conferencing andprocessing module 1200 also includes frame buffers 1305 and 1310, audioprocessing manager 1315, and audio driver 1320. In some embodiments, theaudio processing manager 1315 is implemented as a separate softwaremodule while in other embodiments the audio processing manager 1315 isimplemented as part of the media exchange module 1220.

The audio processing manager 1315 processes audio data captured by thedual camera mobile device for transmission to the other device in thevideo conference. For example, the audio processing manager 1315receives audio data through the audio driver 1320, which is captured bymicrophone 1325, and encodes the audio data before storing the encodedaudio data in the buffer 1305 for transmission to the other device. Theaudio processing manager 1315 also processes audio data captured by andreceived from the other device in the video conference. For instance,the audio processing manager 1315 retrieves audio data from the buffer1310 and decodes the audio data, which is then output through the audiodriver 1320 to the speaker 1330.

In some embodiments, the video conference module 1202 along with theaudio processing manager 1315 and its associated buffers are part of alarger conference module. When a multi-participant audio conference isconducted between several devices without exchange of video content,this video conferencing and processing module 1200 only uses thenetworking manager 1214 and the audio processing manager 1315 tofacilitate the exchange of audio over an Internet Protocol (IP) layer.

The operation of the video conference manager 1204 of some embodimentswill now be described by reference to FIG. 14. FIG. 14 conceptuallyillustrates a process 1400 performed by a video conference manager ofsome embodiments such as video conference manager 1204 illustrated inFIG. 12. This can be equivalent to being performed by the managementlayer 635 of FIG. 6. In some embodiments, the video conference manager1204 performs process 1400 when a user of the dual camera mobile deviceaccepts (e.g., through a user interface displayed on the dual cameramobile device) a video conference request or when a user of anotherdevice accepts a request sent by the user of the dual camera mobiledevice.

The process 1400 begins by receiving (at 1405) instructions to start avideo conference. In some embodiments, the instructions are receivedfrom the client application 1265 or are received from a user through auser interface displayed on the dual camera mobile device and forwardedto the video conference manager 1204 by the client application 1265. Forexample, in some embodiments, when a user of the dual camera mobiledevice accepts a video conference request, the instructions are receivedthrough the user interface and forwarded by the client application. Onthe other hand, when a user of the other device accepts a request sentfrom the local device, some embodiments receive the instructions fromthe client application without user interface interaction (althoughthere may have been previous user interface interaction to send out theinitial request).

Next, the process 1400 initializes (at 1410) a first module thatinteracts with the video conference manager 1204. The modules of someembodiments that interact with the video conference manager 1204 includethe CIPU 1250, the image processing manager 1208, the audio processingmanager 1315, and the networking manager 1214.

In some embodiments, initializing the CIPU 1250 includes instructing theCIPU 1250 to start processing images captured by one or both cameras ofthe dual camera mobile device. Some embodiments initialize the imageprocessing manager 1208 by instructing the image processing manager 1208to start retrieving images from the buffer 1210 and processing andencoding the retrieved images. To initialize the audio processingmanager 1315, some embodiments instruct the audio processing manager1315 to begin encoding audio data captured by the microphone 1325 anddecoding audio data stored in the buffer 1310 (which was received fromthe other device) in order to output to the speaker 1330. Theinitializing of the networking manager 1214 of some embodiments includesinstructing the networking manager 1214 to establish a networkconnection with the other device in the video conference.

The process 1400 then determines (at 1415) whether there are any modulesleft to initialize. When there are modules left to initialize, theprocess 1400 returns to operation 1410 to initialize another of themodules. When all of the required modules have been initialized, theprocess 1400 generates (at 1420) composite images for displaying on thedual camera mobile device (i.e., local display). These composite imagesmay include those illustrated in FIG. 65 in the above-incorporated U.S.patent application Ser. No. ______, entitled “Establishing VideoConference During a Phone Call,” with Attorney Docket Number APLE.P0212,and can include various combinations of images from the cameras of thelocal dual camera mobile device and images from cameras of the otherdevice participating in the video conference.

Next, the process 1400 determines (at 1425) whether a change has beenmade to the video conference. Some embodiments receive changes to thevideo conference through user interactions with a user interfacedisplayed on the dual camera mobile device while other embodimentsreceive changes to the video conference from the other device throughthe networking manager 1214 (i.e., remote control). The changes to videoconference settings may also be received from the client application1265 or other modules in the video conference module 1202 in someembodiments. The video conference settings may also change due tochanges in the network conditions.

When a change has been made, the process 1400 determines (at 1430)whether the change to the video conference is a change to a networksetting. In some embodiments, the changes are either network settingchanges or image capture setting changes. When the change to the videoconference is a change to a network setting, the process modifies (at1440) the network setting and then proceeds to operation 1445. Networksetting changes of some embodiments include changing the bit rate atwhich images are encoded or the frame rate at which the images aretransmitted to the other device.

When the change to the video conference is not a change to a networksetting, the process 1400 determines that the change is a change to animage capture setting and then proceeds to operation 1435. The process1400 then performs (at 1435) the change to the image capture setting. Insome embodiments, change to the image capture settings may includeswitching cameras (i.e., switching which camera on the dual cameramobile device will capture video), focus adjustment, exposureadjustment, displaying or not displaying images from one or both camerasof the dual camera mobile device, and zooming in or out of imagesdisplayed on the dual camera mobile device, among other setting changes.

At operation 1445, the process 1400 determines whether to end the videoconference. When the process 1400 determines to not end the videoconference, the process 1400 returns to operation 1420. When the process1400 determines that the video conference will end, the process 1400ends. Some embodiments of the process 1400 determine to end the videoconference when the process 1400 receives instructions from the clientapplication 1265 to end the video conference (i.e., due to instructionsreceived through the user interface of the local dual camera mobiledevice or received from the other device participating in the videoconference).

In some embodiments, the video conference manager 1204 performs variousoperations when the video conference ends that are not shown in process1400. Some embodiments instruct the CIPU 1250 to stop producing images,the networking manager 1214 to tear down the network connection with theother device in the video conference, and the image processing manager1208 to stop processing and encoding images.

E. Image Processing Manager & Encoder

In addition to temporal noise reduction and image processing operationsperformed by the CIPU and/or CIPU driver, some embodiments perform avariety of image processing operations at the image processing layer 630of the video conference module 625. These image processing operationsmay include exposure adjustment, focus adjustment, perspectivecorrection, adjustment of dynamic range, and image resizing, amongothers.

FIG. 15 conceptually illustrates a process 1500 for performing suchimage processing operations. In some embodiments, some or all of theoperations of the process 1500 are performed by a combination of theimage processing manager 1208 and the encoder driver 1235 of FIG. 12. Insome of such embodiments, the image processing manager 1208 performs thepixel-based processing (e.g., resizing, dynamic range adjustment,perspective correction, etc.). Some embodiments perform process 1500during a video conference on images that are to be transmitted toanother device participating in the video conference.

The process 1500 will now be described by reference to FIG. 12. Theprocess starts by retrieving (at 1505) an image from the buffer 1206. Insome embodiments, the retrieved image is an image of a video (i.e., animage in a sequence of images). This video may have been captured by acamera of a device on which the process 1500 is performed.

Next, the process 1500 performs (at 1510) exposure adjustment on theretrieved image. Some embodiments perform exposure adjustments through auser interface that is displayed on the dual camera mobile device. FIG.16 illustrates an example exposure adjustment operation of suchembodiments.

This figure illustrates the exposure adjustment operation by referenceto three stages 1610, 1615, and 1620 of a UI 1605 of a device 1600. Thefirst stage 1610 illustrates the UI 1605, which includes a display area1625 and a display area 855. As shown, the display area 1625 displays animage 1630 of a sun and a man with a dark face and body. The dark faceand body indicates that the man is not properly exposed. The image 1630could be a video image captured by a camera of the device 1600. Asshown, the display area 855 includes a selectable UI item 1650 forending the video conference. In some embodiments, the layout of thedisplay area 855 is the same as the layout of the display area 855 ofFIG. 9, described above.

The second stage 1615 illustrates a user of the device 1600 initiatingan exposure adjustment operation by selecting an area of the displayarea 1625. In this example, a selection is made by placing a finger 1635anywhere within the display area 1625. In some embodiments, a userselects exposure adjustment from a menu of possible image settingadjustments.

The third stage 1620 shows an image 1640 of the man after the exposureadjustment operation is completed. As shown, the image 1640 is similarto the image 1630, but the man in the image 1640 is properly exposed. Insome embodiments, the properly exposed image is an image that iscaptured after the improperly exposed image. The exposure adjustmentoperation initiated in the second stage 1615 adjusts the exposure ofsubsequent images captured by the camera of the device 1600.

Returning to FIG. 15, the process 1500 next performs (at 1515) focusadjustment on the image. Some embodiments perform focus adjustmentthrough a user interface that is displayed on the dual camera mobiledevice. FIG. 17 conceptually illustrates an example of such focusadjustment operations.

FIG. 17 illustrates a focus adjustment operation by reference to threedifferent stages 1710, 1715, and 1720 of a UI 1705 of a device 1700. Thefirst stage 1710 illustrates the UI 1705 including a display area 1725and a display area 855. The display area 1725 presents a blurry image1730 of a man captured by a camera of the device 1700. The blurrinessindicates that the image 1730 of the man is out of focus. That is, thelens of the camera was not focused on the man when the image 1730 of theman was captured by the camera. Also, the image 1730 could be a videoimage captured by a camera of the device 1700. As shown, the displayarea 855 includes a selectable UI item 1750 for ending the videoconference. In some embodiments, the layout of the display area 855 isthe same as the layout of the display area 855 of FIG. 9, describedabove.

The second stage 1715 illustrates a user of the device 1700 initiating afocus adjustment operation by selecting an area of the display area1725. In this example, a selection is made by placing a finger 1735anywhere within the display area 1725. In some embodiments, a userselects focus adjustment from a menu of possible image settingadjustments.

The third stage 1720 shows an image 1740 of the man after the focusadjustment operation is completed. As shown, the image 1740 is the sameas the image 1730, but the man in the image 1740 appears sharper. Thisindicates that the lens of the camera is properly focused on the man. Insome embodiments, the properly focused image is an image that iscaptured after the improperly focused image. The focus adjustmentoperation initiated in the second stage 1715 adjusts the focus ofsubsequent images captured by the camera of the device 1700.

Back to FIG. 15, the process 1500 performs (at 1520) image resizing onthe image. Some embodiments perform image resizing on the image toreduce the number of bits used to encode the image (i.e., lower the bitrate). In some embodiments, the process 1500 performs image resizing asdescribed by reference to FIG. 26 in the above-incorporated U.S. patentapplication Ser. No. ______, entitled “Establishing Video ConferenceDuring a Phone Call,” with Attorney Docket Number APLE.P0212.

The process 1500 next performs (at 1525) perspective correction on theimage. In some embodiments, the process 1500 performs perspectivecorrection as described in FIG. 24 in the above-incorporated U.S. patentapplication Ser. No. ______, entitled “Establishing Video ConferenceDuring a Phone Call,” with Attorney Docket Number APLE.P0212. Suchperspective correction involves using data taken by one or moreaccelerometer and/or gyroscope sensors that identifies orientation andmovement of the dual camera mobile device. This data is then used tomodify the image to correct for the perspective being off.

After perspective correction is performed on the image, the process 1500adjusts (at 1530) the dynamic range of the image. In some embodiments,the dynamic range of an image is the range of possible values that eachpixel in the image can have. For example, an image with a dynamic rangeof 0-255 can be adjusted to a range of 0-128 or any other range ofvalues. Adjusting the dynamic range of an image can reduce the amount ofbits that will be used to encode the image (i.e., lower the bit rate)and thereby smooth out the image.

Adjusting the dynamic range of an image can also be used for variousother purposes. One purpose is to reduce image noise (e.g., the imagewas captured by a noisy camera sensor). To reduce noise, the dynamicrange of the image can be adjusted so that the black levels areredefined to include lighter blacks (i.e., crush blacks). In thismanner, the noise of the image is reduced. Another purpose of dynamicrange adjustment is to adjust one or more colors or range of colors inorder to enhance the image. For instance, some embodiments may assumethat the image captured by the front camera is an image of a person'sface. Accordingly, the dynamic range of the image can be adjusted toincrease the red and pinks colors to make the person's cheeks appearrosy/rosier. The dynamic range adjustment operation can be used forother purposes as well.

Finally, the process 1500 determines (at 1535) one or more ratecontroller parameters that are used to encode the image. Such ratecontroller parameters may include a quantization parameter and a frametype (e.g., predictive, bi-directional, intra-coded) in someembodiments. The process then ends.

While the various operations of process 1500 are illustrated as beingperformed in a specific order, one of ordinary skill will recognize thatmany of these operations (exposure adjustment, focus adjustment,perspective correction, etc.) can be performed in any order and are notdependent on one another. That is, the process of some embodiments couldperform focus adjustment before exposure adjustment, or similarmodifications to the process illustrated in FIG. 15.

F. Networking Manager

FIG. 18 conceptually illustrates the software architecture of anetworking manager 1800 of some embodiments such as the networkingmanager 1214 illustrated in FIG. 12. As described above, the networkingmanager 1800 manages network connections (e.g., connectionestablishment, connection monitoring, connection adjustments, connectiontear down, etc.) between a dual camera mobile device on which itoperates and a remote device in a video conference. During the videoconference, the networking manager 1800 of some embodiments alsoprocesses data for transmission to the remote device and processes datareceived from the remote device.

As shown in FIG. 18, the networking manager 1800 includes a sessionnegotiating manager 1805, a transmitter module 1815, a universaltransmission buffer 1820, a universal transmission buffer manager 1822,a virtual transport protocol (VTP) manager 1825, a receiver module 1830,and a media transport manager 1835.

The session negotiating manager 1805 includes a protocol manager 1810.The protocol manager 1810 ensures that the transmitter module 1815 usesa correct communication protocol to transmit data to a remote deviceduring the video conference and enforces rules of the communicationprotocol that is used. Some embodiments of the protocol manager 1810support a number of communication protocols, such as a real-timetransport protocol (RTP), a transmission control protocol (TCP), a userdatagram protocol (UDP), and a hypertext transfer protocol (HTTP), amongothers.

The session negotiating manager 1805 is responsible for establishingconnections between the dual camera mobile device and one or more remotedevices participating in the video conference, as well as tearing downthese connections after the conference. In some embodiments, the sessionnegotiating manager 1805 is also responsible for establishing multimediacommunication sessions (e.g., to transmit and receive video and/or audiostreams) between the dual camera mobile device and the remote devices inthe video conference (e.g., using a session initiation protocol (SIP)).

The session negotiating manager 1805 also receives feedback data fromthe media transport manager 1835 and, based on the feedback data,determines the operation of the universal transmission buffer 1820(e.g., whether to transmit or drop packets/frames) through the universaltransmission buffer manager 1822. This feedback, in some embodiments,may include one-way latency and a bandwidth estimation bit rate. Inother embodiments, the feedback includes packet loss information androundtrip delay time (e.g., determined based on packets sent to theremote device in the video conference and the receipt ofacknowledgements from that device). Based on the information from themedia transport manager 1835, the session negotiating manager 1805 candetermine whether too many packets are being sent and instruct theuniversal transmission buffer manager 1822 to have the universaltransmission buffer 1820 transmit fewer packets (i.e., to adjust the bitrate).

The transmitter module 1815 retrieves encoded images (e.g., as abitstream) from a video buffer (e.g., the buffer 1212 of FIG. 12) andpacketizes the images for transmission to a remote device in the videoconference through the universal transmission buffer 1820 and thevirtual transport protocol manager 1825. The manner in which the encodedimages are created and sent to the transmitter module 1815 can be basedon instructions or data received from the media transport manager 1835and/or the session negotiating manager 1805. In some embodiments,packetizing the images involves breaking the received bitstream into agroup of packets each having a particular size (i.e., a size specifiedby the session negotiating manager 1805 according to a particularprotocol), and adding any required headers (e.g., address headers,protocol specification headers, etc.).

The universal transmission buffer manager 1822 controls the operation ofthe universal transmission buffer 1820 based on data and/or instructionsreceived from the session negotiating manager 1805. For example, theuniversal transmission buffer manager 1822 may be instructed to directthe universal transmission buffer 1820 to transmit data, stoptransmitting data, drop data, etc. As described above, in someembodiments when a remote device participating in the conference appearsto be dropping packets, this will be recognized based onacknowledgements received from the remote device. To reduce the packetdropping, the universal transmission buffer manager 1822 may beinstructed to transmit packets at a slower rate to the remote device.

The universal transmission buffer 1820 stores data received from thetransmitter module 1815 and transmits the data to the remote devicethrough the VTP manager 1825. As noted above, the universal transmissionbuffer 1820 may drop data (e.g., images of the video) based oninstructions received from the universal transmission buffer manager1822.

In some embodiments, RTP is used to communicate data packets (e.g.,audio packets and video packets) over UDP during a video conference.Other embodiments use RTP to communicate data packets over TCP duringthe video conference. Other transport layer protocols can be used aswell in different embodiments.

Some embodiments define a particular communication channel between twomobile devices by a pair of port numbers (i.e., source port number anddestination port number). For instance, one communication channelbetween the mobile devices can be defined by one pair of port numbers(e.g., source port 50 and destination port 100) and another differentcommunication channel between the mobile devices can be defined byanother different pair of port numbers (e.g., source port 75 anddestination port 150). Some embodiments also use a pair of internetprotocol (IP) addresses in defining communication channels. Differentcommunication channels are used to transmit different types of datapackets in some embodiments. For example, video data packets, audio datapackets, and control signaling data packets can be transmitted inseparate communication channels. As such, a video communication channeltransports video data packets and an audio communication channeltransports audio data packets.

In some embodiments, a control communication channel is for messagingbetween the local mobile device and a remote device during a videoconference. Examples of such messaging include sending and receivingrequests, notifications, and acknowledgements to such requests andnotifications. Another example of messaging includes sending remotecontrol instruction messages from one device to another. For instance,the remote control operations described in the above-incorporated U.S.patent application Ser. No. ______, entitled “Establishing VideoConference During a Phone Call,” with Attorney Docket Number APLE.P0212(e.g., instructing a device to only send images from one particularcamera or to only capture images with a particular camera) can beperformed by sending instructions from a local device to a remote devicethrough the control communication channel for the local device toremotely control operations of the remote device. Different embodimentsimplement the control communication using different protocols like areal-time transport control protocol (RTCP), an RTP extension, SIP, etc.For instance, some embodiments use RTP extension to relay one set ofcontrol messages between two mobile devices in a video conference anduse SIP packets to relay another set of control messages between themobile devices during the video conference.

The VTP manager 1825 of some embodiments allows different types of datapackets that are specified to be transmitted through differentcommunication channels (e.g., using different pairs of port numbers) tobe transmitted through a single communication channel (e.g., using thesame pair of port numbers). One technique for doing this involvesidentifying the data packet types, identifying the communication channelthrough which data packets are specified to be transmitted by extractingthe specified pair of port numbers of the data packets, and specifyingthe data packets to be transmitted through the single communicationchannel by modifying the pair of port numbers of the data packets to bethe pair of port numbers of the single communication channel (i.e., allthe data packets are transmitted through the same pair of port numbers).

To keep track of the original pair of port numbers for each type of datapacket, some embodiments store a mapping of the original pair of portnumbers for the data packet type. Some of these embodiments than use thepacket type field of the protocol to differentiate the different packetsthat are being multiplexed into one communication channel. For instance,some embodiments that have the VTP manager multiplex audio, video andcontrol packets into one RTP stream, use the RTP packet type field todifferentiate between the audio, video and control packets that aretransmitted in the one RTP channel to the other device in the videoconference. In some of these embodiments, the VTP manger also routescontrol messaging in SIP packets to the other device.

Some embodiments identify examine the data packet signatures (i.e.,packet header formats) to distinguish between different packets that arecommunicated using different protocols (e.g., to differentiate betweenpackets transported using RTP and packets transported using SIP). Insuch embodiments, after the data packets of the different protocols aredetermined, the fields of the data packets that use the same protocol(e.g., audio data and video data using RTP) are examined as describedabove to identify the different data types. In this manner, the VTPmanager 1825 transmits different data packets, which are intended to betransmitted through different communication channels, through a singlecommunication channel.

Although one way of combining different types of data through a singlecommunication channel is described above, other embodiments utilizeother techniques to multiplex different packet types into onecommunication stream. For example, one technique of some embodimentsinvolves keeping track of the original pair of port numbers of the datapackets and storing the original pair of port numbers in the data packetitself to be later extracted. Still other ways exist for combiningdifferent types of data between two video conference participants intoone port pair channel.

When the VTP manager 1825 receives data packets from the remote devicethrough a virtualized communication channel, the VTP manager 1825examines the signatures of the data packets to identify the differentpackets that are sent using the different protocols. Such signatures canbe used to differentiate SIP packets from RTP packets. The VTP managerof some embodiments also uses the packet type field of some or all ofthe packets to demultiplex the various different types of packets (e.g.,audio, video and control packets) that were multiplexed into a singlevirtualized channel. After identifying these different types of packets,the VTP manager associates each different type of packet with itscorresponding port pair numbers based on a mapping of port pair numbersand packet types that it keeps. The VTP manager 1825 then modifies thepair of port numbers of the data packets with the identified pair ofport numbers and forwards the data packets to be depacketized. In otherembodiments that use different techniques for multiplexing differentpacket types into the single channel, the VTP manager uses differenttechniques for parsing out the packets.

By using such techniques for multiplexing and de-multiplexing thedifferent packets, the VTP manager 1825 creates a single virtualizedcommunication channel (e.g., a single pair of port numbers), transmitsthe video data, audio data, and control signaling data through thesingle virtualized communication channel, and receives audio, video, andcontrol packets from the remote device through the single virtualizedcommunication channel. Thus, from the perspective of the network, datais transmitted through this single virtualized communication channel,while, from the perspective of the session negotiating manager 1805 andthe protocol manager 1810, the video data, audio data, and controlsignaling data are transmitted through different communication channels.

Similar to the images that are transmitted to the remote device in thevideo conference, images transmitted from the remote device in the videoconference are received in packet format. The receiver module 1830receives the packets and depacketizes them in order to reconstruct theimages before storing the images in a video buffer (e.g., the buffer1216 of FIG. 12) to be decoded. In some embodiments, depacketizing theimages involves removing any headers and reconstructing a bitstream thatonly has image data (and potentially size data) from the packets.

The media transport manager 1835 processes feedback data (e.g., one-waylatency, bandwidth estimation bit rate, packet loss data, roundtripdelay time data, etc.) received from the network to dynamically andadaptively adjust the rate of data transmission (i.e., bit rate). Themedia transport manager 1835 also controls error resilience based on theprocessed feedback data in some other embodiments, and may also send thefeedback data to the video conference manager 1204 in order to adjustother operations of the video conference module 1202 such as scaling,resizing, and encoding. In addition to having the universal transmissionbuffer drop packets when a remote device in the conference is not ableto process all of the packets, the video conference module and encodercan use a lower bit rate for encoding the images so that fewer packetswill be sent for each image.

In some embodiments, the media transport manager 1835 may also monitorother variables of the device such as power consumption and thermallevels that may affect how the operational power modes of the camerasare configured, as discussed above. This data may also be used asadditional inputs into the feedback data (e.g., if the device is gettingtoo hot, the media transport manager 1835 may try to have the processingslowed down).

Several example operations of the networking manager 1800 will now bedescribed by reference to FIG. 12. The transmission of images capturedby a camera of the dual camera mobile device to a remote device in thevideo conference will be described first, followed by the description ofreceiving images from the remote device. The transmitter module 1815retrieves encoded images from the buffer 1212, which are to betransmitted to the remote device in the video conference.

The protocol manager 1810 determines the appropriate protocol to use(e.g., RTP to transmit audio and video) and the session negotiatingmanager 1805 informs the transmitter module 1815 of such protocol. Next,the transmitter module 1815 packetizes the images and sends thepacketized images to the universal transmission buffer 1820. Theuniversal transmission buffer manager 1822 receives instructions fromthe session negotiating manager 1805 to direct the universaltransmission buffer 1820 to transmit or drop the images. The VTP manager1825 receives the packets from the universal transmission buffer 1820and processes the packets in order to transmit the packets through asingle communication channel to the remote device.

When receiving images from the remote device, the VTP manager 1825receives packetized images from the remote device through thevirtualized single communication channel and processes the packets inorder to direct the images to the receiver module 1830 through acommunication channel that is assigned to receive the images (e.g., avideo communication channel).

The receiver module 1830 depacketizes the packets to reconstruct theimages and sends the images to the buffer 1216 for decoding by thedecoder 1260. The receiver module 1830 also forwards control signalingmessages to the media transport manager 1835 (e.g., acknowledgements ofreceived packets from the remote device in the video conference).

Several example operations of the networking manager 1800 were describedabove. These are only illustrative examples, as various otherembodiments will perform these or different operations using differentmodules or with functionalities spread differently between the modules.Furthermore, additional operations such as dynamic bit rate adjustmentmay be performed by the modules of networking manager 1800 or othermodules.

IV. In-Conference Adjustment and Control Operations

A. Picture-in-Picture Modifications

1. Rotate

Some embodiments rotate the PIP display that is presented during a videoconference when a user of the mobile device used for the videoconference rotates the device during the conference. FIG. 19 illustratesthe rotation of a UI 805 of a device 1900 when the device is rotatedfrom a vertical position to a horizontal position. The device 1900 isheld vertically when the long side of the screen is vertical whereas thedevice 1900 is held horizontally when the long side of the screen ishorizontal. In the example illustrated in FIG. 19, the UI 805 rotatesfrom a portrait view that is optimized for a vertical holding of thedevice to a landscape view that is optimized for horizontal holding ofthe device 1900. This rotation functionality allows the user to view theUI 805 displayed in an upright position when the mobile device 1900 isheld either vertically or horizontally.

FIG. 19 illustrates the rotation of the UI 805 in terms of six differentoperational stages 1910, 1915, 1920, 1925, 1930 and 1935. The firststage 1910 illustrates the UI 805 during a video conference between thelocal user of the device and a remote user of a remote device. The UI805 in FIG. 19 shows a PIP display 880 that is the same PIP displayshown in the fifth stage of FIG. 8 after the video conference has beenestablished. In this example, the video captured by the local user'sdevice is displayed in the inset display area 860 and the video capturedby the remote user's device is displayed in the background display area870. In the display area 855 below the PIP display 880 includes aselectable UI item 1985 (e.g., an End Conference button 1985), which theuser may select to end the video conference (e.g., through a singlefinger tap).

The second stage 1915 illustrates the UI 805 after the user begins totilt the device 1900 sideways. In this example, the user has started totilt the device 1900 from being held vertically to being heldhorizontally, as indicated by the arrow 1960. The appearance of the UI805 has not changed. In other situations, the user may want to tilt thedevice 1900 from being held horizontally to being held verticallyinstead, and, in these situations, the UI 805 switches from ahorizontally optimized view to a vertically optimized view.

The third stage 1920 illustrates the UI 805 in a state after the device1900 has been tilted from being held vertically to being heldhorizontally. In this state, the appearance of the UI 805 still has notchanged. In some embodiments, the rotation operation is triggered afterthe device 1900 is tilted past a threshold amount and is kept past thispoint for a duration of time. In the example illustrated in FIG. 19, itis assumed that the threshold amount and the speed of the rotation donot cause the UI 805 to rotate until a short time interval after thedevice has been placed in the horizontal position. Different embodimentshave different threshold amounts and waiting periods for triggering therotation operation. For example, some embodiments may have such a lowthreshold to triggering the rotation operation as to make the UI 805appear as if it were always displayed in an upright position,notwithstanding the orientation of the device 1900. In otherembodiments, the user of the device 1900 may specify when the rotationoperation may be triggered (e.g., through a menu preference setting).Also, some embodiments may not delay the rotation after the device istilted past the threshold amount. Moreover, different embodiments mayallow the rotation operation to be triggered in different ways, such asby toggling a switch on the mobile device, by giving voice commands,upon selection through a menu, etc.

The fourth stage 1925 illustrates the UI 805 after the rotationoperation has started. Some embodiments animate the rotation displayareas to provide feedback to the user regarding the rotation operation.FIG. 19 illustrates an example of one such animation. Specifically, itshows in its fourth stage 1925 the start of the rotation of the displayareas 880 and 855 together. The display areas 880 and 855 rotate aroundan axis 1965 going through the center of the UI 805 (i.e., the z-axis).The display areas 880 and 855 are rotated the same amount but in theopposite direction of the rotation of the device 1900 (e.g., through thetilting of the device 1900). In this example, since the device 1900 hasrotated ninety degrees in a clockwise direction (by going from beingheld vertically to being held horizontally) the rotation operation wouldcause the display areas 880 and 855 to rotate ninety degrees in acounter clockwise direction. As the display areas 880 and 855 rotate,the display areas 880 and 855 shrink proportionally to fit the UI 805 sothat the display areas 880 and 855 may still appear entirely on the UI805. Some embodiments may provide a message to indicate the state ofthis device 1900 (e.g., by displaying the word “Rotating”).

The fifth stage 1930 illustrates the UI 805 after the display areas 880and 855 have rotated ninety degrees counter clockwise from portrait viewto landscape view. In this stage, the display areas 880 and 855 havebeen rotated but have not yet expanded across the full width of the UI805. The arrows 1975 indicate that at the end of the fifth stage, thedisplay areas 880 and 855 will start to laterally expand to fit the fullwidth of the UI 805. Different embodiments may not include this stagesince the expansion could be performed simultaneously with the rotationin the fourth stage 1925.

The sixth stage 1935 illustrates the UI 805 after the display areas 880and 855 have been expanded to occupy the full display of the UI 805. Asmentioned above, other embodiments may implement this rotationdifferently. For some embodiments, simply rotating the screen of adevice past a threshold amount may trigger the rotation operation,notwithstanding the orientation of the device 1900.

Also, other embodiments might provide a different animation forindicating the rotation operation. The rotation operation performed inFIG. 19 involves the display areas 880 and 855 rotating about the centerof the UI 805. Alternatively, the display areas may be individuallyrotated about the center axis of their individual display areas. Onesuch approach is shown in FIG. 20. FIG. 20 shows an alternative methodto animating the rotation of the display areas 870 and 860 of PIPdisplay 880 of a UI 805. The PIP display 880 illustrated in FIG. 20 isthe same PIP display 880 illustrated in FIG. 8.

FIG. 20 illustrates the rotation of the PIP display 880 in terms of sixdifferent operational stages 1910, 1915, 1920, 2025, 2030, and 2035. Thefirst three stages of operation of the UI 805 are identical to the firstthree stages of operation as described in the UI 805 in FIG. 19. At thethird stage for both FIGS. 19 and 20, the device 2000 has gone frombeing held vertically to being held horizontally and the rotation of theUI 805 has not yet begun.

The fourth stage 2025 illustrates the alternative method to animatingthe rotation. In this stage, the rotation operation has started.Specifically, the fourth stage shows 2025 the start of the rotation ofthe display areas 870 and 860. The display areas 870 and 860 each rotatearound axes 2067 and 2065, respectively, going through the center ofeach of the display areas (i.e., the z-axis). The display areas 870 and860 are rotated the same amount but in the opposite direction of therotation of the device 2000 (e.g., through the tilting of the device2000). Similar to that illustrated in the fourth stage 1925 of FIG. 19above, since the device 2000 has rotated ninety degrees in a clockwisedirection (by going from being held vertically to being heldhorizontally) the rotation operation would cause the display areas 870and 860 to rotate ninety degrees in a counter clockwise direction. Asthe display areas 870 and 860 rotate, the display areas 870 and 860shrink proportionally to fit the UI 805 so that the display areas 870and 860 may still appear entirely on the UI 805.

The fifth stage 2030 illustrates the UI 805 after each of the displayareas 870 and 860 have rotated ninety degrees counter clockwise fromportrait view to landscape view. In this stage, the display areas 870and 860 have been rotated but have not yet expanded across the fullwidth of the UI 805. Moreover, the display area 860 has not moved intoits final position. The final position of the inset display area 860 inthe PIP display 880 is determined by the position of the inset displayarea 860 in the PIP display 880 as shown in the first stage 1910 (e.g.,the inset display area 860 in the lower left corner of the PIP display880). In this stage, the inset display area 860 is still in the upperleft corner of the UI 805.

The arrows 2080 indicate that at the end of the fifth stage 2030, thedisplay areas 870 and 860 will start to laterally expand until the maindisplay area 870 fits the full width of the UI 805 for a device that isheld horizontally. Moreover, the arrow 2075 indicates that the insetdisplay area 860 will slide to the lower left corner of the PIP display880.

Different embodiments may implement this differently. In someembodiments, the moving of the inset display area 860 may occursimultaneously as the expansion of the main display area 870 orsequentially. Moreover, some embodiments may resize the inset displayareas 860 before, during or after the expansion of the main display area870 to create the new PIP display 880. In this example, the display area855 disappears while the display areas 860 and 870 are rotating.However, the display area 855 may remain on the UI 805 during therotation and rotate along with the display areas 860 and 870 in someembodiments.

The sixth stage 2035 illustrates the UI 805 after the inset display area860 has reached its new location and the display areas 860 and 870 havebeen properly expanded to fit the full width of the UI 805. In thisexample, the inset display area 860 is now in the lower left corner ofthe PIP display 880, overlapping the main display area 870. The PIPdisplay 880 now has the same display arrangement as the PIP display 880from the first stage 1910. The appearance of the display area 855 belowthe PIP display 880 in the sixth stage indicates that the rotationoperation is completed. As noted above, simply rotating the screen of adevice past a threshold amount may trigger the rotation operation,notwithstanding the orientation of the device 2000.

In the examples described above by reference to FIGS. 19 and 20, theorientation of the display area 870 also changes (i.e., from portrait tolandscape). That is, after the display area 870 is rotated in the thirdstage 1920, the orientation of the display area 870 changes fromportrait to landscape by horizontally expanding the PIP display 880 sothat it fills the entire UI 805. In some embodiments, when the device2000 is rotated, video captured by the remote device rotates but theorientation of the display area that displays the video captured by theremote device remains unchanged. One such example is illustrated in FIG.21. This figure is similar to FIG. 20 except that video displayed in thedisplay area 870 rotates but the display area 870 remains displayed inportrait orientation.

FIG. 21 also illustrates an example of a rotation operation in which thedisplay area 855 remains in the same position (instead of rotating andexpanding horizontally to fill the PIP display 880 as shown in FIG. 20).Moreover, this figure includes a layout of the display area 855 that isthe same as the layout of the display area 855, described above in FIG.9. As shown, the display area 855 remains in the same position as thedevice 2000 rotates in the stages 2140, 2145, 2150, 2155, 2185, and2190.

Some embodiments provide a rotation operation in which the orientationof the display area that displays video captured by the local devicechanges (instead of remaining in the same orientation as shown in FIG.20) to reflect the orientation of the local device after the rotationoperation is performed on the local device. FIG. 21 illustrates anexample of such a rotation operation of a UI 805 by reference to sixdifferent stages 2140, 2145, 2150, 2155, 2185, and 2190. In this figure,the first stage 2140 shows the inset display area 860, which displaysvideo captured by a camera of the device 2000, in a portraitorientation. The second and third stages 2145 and 2150 are similar tothe second and third stages 1915 and 1920 of FIG. 20 as they show thetilting of the device 2000 at various stages of the rotation operation.At this point, the camera of the device 2000 is capturing images in alandscape orientation. To indicate this transition, some embodimentsprovide an animation as shown in fourth and fifth stages 2155 and 2185while other embodiments do not provide any animation at all.

In the fourth stage 2155, the image displayed in the inset display area860 is rotated, but not the inset display area 860 itself since thetilting of the device 2000 in the second and third stages 1945 and 2150has rotated the inset display area 860 to a landscape orientation. Inthe fifth stage 2185, the rotated image in the inset display area 860 ishorizontally expanded to fill the inset display area 860 and the insetdisplay area 860 starts to move towards the lower left area of the PIPdisplay 880 to position the inset display area 860 in the same relativeposition as the inset display area 860 in the PIP display of the firststage 2140.

In some embodiments, the orientation of the display area that displaysthe video captured by the remote device also changes to reflect theorientation of the remote device after a rotation operation is performedon the remote device. FIG. 22 illustrates four different stages of a UI805 of the device 2000 in which (1) the orientation of the display areathat displays the video captured by the local device (display area 860in this example) changes to reflect the orientation of the local deviceafter a rotation operation is performed on the local device and (2) theorientation of the display area that displays video captured by theremote device (display area 870 in this example) changes to reflect theorientation of the remote device after a rotation operation is performedon the remote device.

In the first stage 2205, the UI 805 is the same as the UI 805 in FIG.21. Specifically, the first stage 2205 shows the display areas 860 and870 in a portrait orientation because the device 2000 is shown in aportrait orientation and the remote device is in a portrait orientation(not shown). From the first stage 2205 to the second stage 2210, arotation operation is performed on the local device by rotating thedevice 2000 ninety degrees from an upright position to a sidewaysposition. The second stage 2210 shows the UI 805 after the rotationoperation of the device 2000 is completed. In this stage, the videosdisplayed in the display areas 870 and 860 have rotated to an uprightposition. However, only the display area 860 of the locally capturedvideo has rotated from a portrait orientation to a landscape orientationsince the rotation operation is only performed on the local device(i.e., the device 2000). The display area 870 remains in the portraitorientation.

From the second stage 2210 to the third stage 2215, a rotation operationis performed on the remote device by rotating the remote device from anupright position to a sideways position (not shown). The third stage2215 shows the UI 805 after the rotation operation of the remote deviceis completed. In this stage, the video displayed in the display area 870and the display area 870 of the remotely captured video have rotatedfrom a portrait orientation to a landscape orientation since therotation operation is only performed on the remote device. Thus, thisstage of the UI 805 displays the display areas 870 and 860 of thelocally and remotely captured videos both in landscape orientation.

From the third stage 2215 to the fourth stage 2220, a rotation operationis performed on the local device by rotating the device 2000 ninetydegrees from a sideways position to an upright position. The fourthstage 2220 shows the UI 805 after the completion of this rotationoperation. In this fourth stage 2220, the videos displayed in thedisplay areas 860 and 870 have rotated to an upright position. However,only the display area 860 of the locally captured video has rotated froma landscape orientation to a portrait orientation since the rotationoperation is only performed on the local device (i.e., the device 2000).The display area 870 remains in the landscape orientation.

From the fourth stage 2220 to the first stage 2205, a rotation operationis performed on the remote device by rotating the remote device ninetydegrees from a sideways position to an upright position (not shown). Inthis case, the first stage 2205 shows the display area 870 after thecompletion of this rotation operation. Therefore, the UI 805 of thisstage shows the display areas 860 and 870 in a portrait orientation.Although FIG. 22 illustrates a sequence of different rotationoperations, other embodiments can perform any number of rotationoperations in any number of different sequences.

FIGS. 19, 20, 21, and 22 describe rotate operations performed on localand remote devices during a video conference. When a rotate operation isperformed on the local mobile device, some embodiments notify the remotedevice of the rotate operation in order for the remote device to performany modifications to the local device's video (such as rotating thedisplay area that is displaying the local device's video). Similarly,when a rotate operation is performed on the remote device, the remotedevice notifies the local device of this operation to allow the localdevice to perform any modifications the remote device's video. Someembodiments provide a control communication channel for communicatingthe notification of rotate operations between the local and remotedevices during the video conference.

Even though FIGS. 19, 20, 21, and 22 illustrate different manners inwhich the animation of a rotation can be performed, one of ordinaryskill will realize that other embodiments may display the animation ofthe rotation in other different ways. In addition, the animation of therotation operation can cause changes to the image processing operationsof the local mobile device such as causing the video conference manager1204 to re-composite the display area(s) at different angles in the UI805 and scale the images displayed in the display area(s).

2. Identifying Regions of Interest

Some embodiments allow a user to identify a region of interest (ROI) ina displayed video during a video conference in order to modify the imageprocessing (e.g., the image processing manager 1208 in FIG. 12), theencoding (e.g., the encoder 1255 in FIG. 12), the behavior of the mobiledevices and their cameras during the video conference, or a combinationthereof. Different embodiments provide different techniques foridentifying such a region of interest in a video. FIG. 23 illustrates auser interface of some embodiments for identifying a region of interestin a video in order to improve the image quality of the video.

In FIG. 23, a UI 2300 of a mobile device 2325 presents a PIP display2365 during a video conference with a remote user of another mobiledevice. The PIP display in FIG. 23 includes two video displays: abackground main display 2330 and a foreground inset display 2335. Inthis example, the background main display 2330 presents a video of atree and a person with a hat, which are assumed to be a tree and aperson whose video is being captured by the remote device's front cameraor a tree and a person whose video is being captured by the remotedevice's back camera. The foreground inset display 2335 presents a videoof a man, which in this example is assumed to be a man whose video isbeing captured by the local device's front camera or a person whosevideo is being captured by the local device's back camera. Below the PIPdisplay is a display area 855 that includes a selectable UI item 2360labeled “End Conference” (e.g. a button 2360) that allows the user toend the video conference by selecting the item.

This PIP display is only one manner of presenting a composite view ofthe videos being captured by the remote and local devices. Someembodiments may provide other composite views. For instance, instead ofhaving a larger background display for the video from the remote device,the larger background display can be of the video from the local deviceand the smaller foreground inset display can be of the video from theremote device. Also, some embodiments allow the local and remote videosto appear in the UI in two side-by-side display areas (e.g. left andright display windows, or top and bottom display windows) or twodiagonally aligned display areas. In other embodiments, the PIP displaymay also contain a larger background display and two smaller foregroundinset displays. The manner of the PIP display or a default display modemay be specified by the user in some embodiments.

FIG. 23 illustrates the ROI identification operation in terms of fouroperational stages of the UI 2300. As shown in the first stage 2305, thevideo presented in the background display 2330 has very low quality(i.e., the video images are fuzzy). In this example, a user of a mobiledevice 2325 would like to identify the area in the background display2330 where the person's face 2370 appears as the region of interest.

In the second stage 2310, the operation of identifying a region ofinterest is initiated. In this example, the operation is initiated byselecting an area in the video presented in the background display 2330that the user wants to identify as the region of interest (e.g., bytapping a finger 2350 on the device's screen at a location about thedisplayed person's face 2370 in the background display 2330).

As shown in the third stage 2315, the user's selection of the areacauses the UI 2300 to draw an enclosure 2375 (e.g., a dotted square2375) surrounding the area of the user's selection. The fourth stage2320 displays the UI 2300 after the identification of the region ofinterest has been completed. As a result of this process, the quality ofthe video within the region of interest has been substantially improvedfrom that in the first stage 2305. The removal of the enclosure 2375indicates that the ROI selection operation is now completed. In someembodiments, the ROI identification process also causes the same changesto the same video displayed on the remote device as it does to the localdevice 2325. In this example for instance, the picture quality withinthe region of interest of the same video displayed on the remote deviceis also substantially improved.

In some embodiments, the user may enlarge or shrink the enclosure 2375in the third stage 2315 (e.g., by holding the finger 2350 down on thedisplay and moving the finger 2350 toward the upper right corner of thescreen to enlarge the enclosure 2375 or moving the finger 2350 towardthe lower left corner of the screen to shrink the enclosure 2375). Someembodiments also allow the user to relocate the enclosure 2375 in thethird stage 2315 (e.g., by holding the finger 2350 down on the displayand moving the finger 2350 horizontally or vertically on the display).In some other embodiments, the selection of the area may not cause theUI 2300 to draw the enclosure 2375 at all in the third stage 2315.

Other embodiments provide different techniques for allowing a user toidentify a region of interest in a video. FIG. 24 illustrates one suchother technique. In FIG. 24, the user identifies a region of interest bydrawing a shape that bounds the region. The shape in this example is arectangle, but it can be other shapes (e.g., any other polygon, acircle, an ellipse, etc.). Some embodiments provide this alternativetechnique of FIG. 24 in a device UI that also provides the techniqueillustrated in FIG. 23. Other embodiments, however, do not provide boththese techniques in the same UI.

FIG. 24 illustrates this ROI identification operation in terms of fiveoperational stages of a UI 2300. The first stage 2305 in FIG. 24 isidentical to the first stage 2305 in FIG. 23. Specifically, in thisfirst stage 2305, the UI 2300 illustrates a PIP display 2365 with alarger background main display 2330 and a smaller foreground insetdisplay 2335 at the bottom left corner of the PIP display 2365.

In the second stage 2410, the operation of identifying a region ofinterest is initiated. In this example, the operation is initiated byselecting for a duration of time a first position for defining theregion of interest in a video presented in the background display area2330 (e.g., by holding a finger 2450 down on the device's screen at alocation about the displayed person's face 2370 in the backgrounddisplay 2330 for a duration of time). In the third stage 2415, the UI2300 indicates that the first position 2470 has been selected in termsof a dot 2455 next to the selected first position on the backgrounddisplay area 2330.

The fourth stage 2420 illustrates the UI 2300 after the user hasselected a second position 2475 for defining the region of interest. Inthis example, the user selects this second position 2475 by dragging thefinger 2450 across the device's screen from the first location after thedot 2455 appears and stopping at a location between the displayed hatand the displayed tree in the background display area 2330, as indicatedby an arrow 2460. As shown in the fourth stage, this dragging caused theUI 2300 to draw a rectangular border 2465 for the region of interestarea that has the first and second positions 2470 and 2475 at itsopposing vertices.

The fifth stage 2425 illustrates the UI 2300 after identification of theregion of interest has been completed. In this example, the usercompletes identification of the region of interest by stopping thedragging of the finger 2450 and removing the finger 2450 from thedevice's display screen once the desired region of interest area hasbeen identified. The fifth stage 2425 illustrates that as a result ofthe drawing process, the quality of the video within the region ofinterest has been substantially improved from that in the first stage2305. In some embodiments, the drawing process also causes the samechanges to the display on the remote device as it does to the localdevice 2325. In this example for instance, the picture quality withinthe region of interest of the same video displayed on the remote devicewill be substantially improved.

The description of FIGS. 23 and 24, above, illustrates different mannersof identifying a region of interest in a video in order to improve thepicture quality of the identified region. In some embodiments, improvingthe picture quality of the identified region of interest causes changesto the encoding operations of the dual camera mobile device such asallocating more bits to the identified region when encoding the video.

Some embodiments allow the user to identify a region of interest in avideo to make different changes to the mobile devices or their cameras.For instance, FIG. 25 illustrates an example of identifying a region ofinterest in a video to expand or shrink the region of interest area onthe display. In this approach, the user identifies a region of interestin a video by selecting an area on the display as the center of theregion of interest and then expanding or shrinking the region ofinterest area.

In FIG. 25, a UI 2500 of a mobile device 2525 presents a PIP display2365 during a video conference with a remote user of another mobiledevice. The PIP display 2365 in FIG. 25 is substantially similar to thePIP display 2365 of FIG. 23, but the foreground inset display 2335 ofFIG. 25 is located in the lower left corner of the PIP display 2365.

FIG. 25 illustrates the ROI selection operation in terms of fouroperational stages of the UI 2500. As shown in the first stage 2505, thebackground display 2530 presents a video with a man on the left and atree 2540 on the right of the display 2530. Moreover, the tree 2540 isrelatively small and occupies only the right side of the backgrounddisplay area 2530. In this example, a user of a mobile device 2525 wouldlike to identify the area where the tree 2540 appears on the display2530 as the region of interest.

In the second stage 2510, the operation of identifying a region ofinterest is initiated. In this example, the operation is initiated byselecting an area 2540 in the video presented in the background display2530 that the user wants to identify as the region of interest (e.g., byholding two fingers 2545 and 2546 down on the background display area2530 where the tree 2540 is displayed). At this stage 2510, the user canmake the region of interest area 2540 expand and take a larger portionof the background display area 2530 by dragging his fingers 2545 and2546 farther away from each other. The user can also make the region ofinterest 2540 shrink to take a smaller portion of the background displayarea 2530 by dragging his fingers 2545 and 2546 closer together.

The third stage 2515 illustrates the UI 2500 after the user has startedto make the region of interest 2540 expand to take up a larger portionof the background display area 2530 by moving his fingers 2545 and 2546farther away from each other (i.e., the finger 2545 moves toward theupper left corner of the background display area 2530 and the finger2546 moves toward the lower right corner of the display 2530), asindicated by arrows 2550. In some embodiments, the finger movement alsocauses the same changes to the display of the remote device as it doesto the local device. In this example for instance, the region ofinterest of the same video will expand and take up a larger portion ofthe background display area 2530 of the remote device. In someembodiments, the expansion of the region of interest in the localdisplay and/or remote display causes one or both of the mobile devicesor their cameras to modify one or more of their other operations, asfurther described below.

The fourth stage 2520 displays the UI 2500 after the identification ofthe region of interest has been completed. In this example, the usercompletes the identification of the region of interest by stopping thedragging of his fingers 2545 and 2546 and removing the fingers 2545 and2546 from the device's display screen once the region of interest hasreached the desired proportion in the background display area 2530. As aresult of this process, the region of interest has taken up a majorityof the background display 2530. The identification of the region ofinterest operation is now completed.

Some of the examples above illustrate how a user may identify a regionof interest in a video for improving the image quality within theselected region of interest in the video (e.g., by increasing the bitrate for encoding the region of interest portion of the video). In someembodiments, identifying a region of interest in the video causeschanges to the image processing operations of the mobile device such asexposure, scaling, focus, etc. For example, identifying a region ofinterest in the video can cause the video conferencing manager 1204 toscale and composite the images of the video differently (e.g.,identifying a region of interest to which to zoom).

In other embodiments, identifying a region of interest in the videocauses changes to the operation of the mobile device's camera(s) (e.g.,frame rate, zoom, exposure, scaling, focus, etc.). In yet otherembodiments, identifying a region of interest in the video causeschanges to the encoding operations of the mobile device like allocatingmore bits to the identified region, scaling, etc. In addition, while theexample ROI identification operations described above may cause only oneof the above-described modifications to the mobile device or itscameras, in some other embodiments the ROI identification operation maycause more than one of the modifications to the operation of the mobiledevice or its cameras. In addition, in some embodiments, the layout ofthe display area 855 in FIGS. 23-25 is the same as the layout of thedisplay area 855 of FIG. 9, described above.

B. Switch Camera

Some embodiments provide procedures to switch cameras (i.e., change thecamera by which images are captured) during a video conference.Different embodiments provide different procedures for performing theswitch camera operation. Some embodiments provide procedures performedby a dual camera mobile device for switching cameras of the device(i.e., local switch) while other embodiments provide procedures for thedual camera mobile device to instruct another dual camera mobile devicein the video conference to switch cameras of the other device (i.e.,remote switch). Yet other embodiments provide procedures for both.Section IV.B.1 will describe a process for performing a local switchcamera operation on a dual camera mobile device. Section IV.B.2 willdescribe a process for performing a remote switch camera operation onthe dual camera mobile device.

1. Local Switch Camera

FIG. 26 illustrates a process 2600 that some embodiments perform on alocal dual camera mobile device to switch between the two cameras of thedevice during a video conference with a remote mobile device thatincludes at least one camera. In some embodiments, the process 2600 isperformed by the video conference manager 1204 shown in FIG. 12. Forpurposes of explanation, the discussion will refer to one camera of thelocal dual camera mobile device as camera 1 and the other camera of thelocal dual camera mobile device as camera 2.

The process 2600 begins by starting (at 2605) a video conference betweenthe local dual camera mobile device and the remote mobile device. Next,the process 2600 sends (at 2610) a video image from the currentlyselected camera (e.g., the camera 1) of the local dual camera mobiledevice to the remote mobile device for display on the remote mobiledevice. At 2610, the process also generates and displays a compositedisplay based on this video image and the video image that it receivesfrom the remote mobile device.

The process 2600 then determines (at 2615) whether a request to end thevideo conference is received. As described above, a video conference canend in some embodiments at the request of a user of the local dualcamera mobile device (e.g., through a user interface of the local dualcamera mobile device) or a user of the remote mobile device (e.g.,through a user interface of the remote mobile device). When the process2600 receives a request to end the video conference, the process 2600ends.

When the process 2600 does not receive a request to end the videoconference, the process 2600 then determines (at 2620) whether the userof the local dual camera mobile device has directed the device to switchcameras for the video conference. The process 2600 returns to operation2610 when the process 2600 determines (at 2620) that it has not beendirected to switch cameras. However, when the process 2600 determines(at 2620) that it has been so directed, the process 2600 transitions to2625.

At 2625, the process 2600 sends a notification to the remote mobiledevice to indicate that the local dual camera mobile device is switchingcameras. In some embodiments, the process 2600 sends the notificationthrough the video conference control channel that is multiplexed withthe audio and video channels by the VTP Manager 1825 as described above.

After sending its notification, the process 2600 performs (at 2630) aswitch camera operation. In some embodiments, performing (at 2630) theswitch camera operation includes instructing the CIPU to stop capturingvideo images with the camera 1 and to start capturing video images withthe camera 2. These instructions can simply direct the CIPU to switchcapturing images from the pixel array associated with the camera 2 andto start processing these images. Alternatively, in some embodiments,the instructions to the CIPU are accompanied by a set of initializationparameters that direct the CIPU (1) to operate the camera 2 based on aparticular set of settings, (2) to capture video generated by the camera2 at a particular frame rate, and/or (3) to process video images fromthe camera 2 based on a particular set of settings (e.g., resolution,etc.).

In some embodiments, the switch camera instruction (at 2630) alsoincludes instructions for switching the unused camera to the fourthoperational power mode as described above. In this example, the switchcamera instructions include instructions for the camera 2 to switch toits fourth operational power mode. In addition, the switch camerainstructions also include instructions for the camera 1 to switch fromits fourth operational power mode to another operational power mode suchas the first operational power mode to conserve power or to the thirdoperational power mode so it can quickly switch to the fourthoperational power mode and start capturing images when requested to doso. The switch camera operation 2630 also involves compositing imagescaptured by the camera 2 of the local dual camera mobile device (insteadof images captured by the camera 1) with images received from the remotemobile device for display on the local dual camera mobile device.

After directing the switch camera at 2630, the process 2600 performs (at2635) a switch camera animation on the local dual camera mobile deviceto display a transition between the display of images from the camera 1and the display of images from the camera 2. Following the switch cameraanimation on the local dual camera mobile device, the process 2600 loopsback through operations 2610-2620 until an end video conference requestor a new switch camera request is received.

FIG. 27 illustrates one example of how some embodiments allow a switchcamera operation to be requested through a UI 805 of a dual cameradevice and how these embodiments animate the switch camera operation.This figure illustrates the switch camera operation in terms of eightdifferent operational stages 2710, 2715, 2720, 2725, 2730, 2735, 2740,and 2745 of the UI 805 of the device. The first four stages 2710, 2715,2720, and 2725 of the UI 805 illustrate an example of receiving a user'srequest to switch cameras. The user of the device has other mechanismsto make such a request in some embodiments of the invention.

The first stage 2710 is the same as the fifth stage 830 of the UI 805 ofFIG. 8, which shows the UI 805 after a video conference is set up. Atthis stage, the UI 805 displays a PIP display that includes two videodisplays: a larger background display from the remote camera and asmaller foreground inset display from the local camera. In this example,the background main display area 870 presents a video of a lady, whichin this example is assumed to be a lady whose video is being captured bythe remote device, while the foreground inset display area 860 presentsa video of a man, which in this example is assumed to be a man whosevideo is being captured by the local device's front camera.

The second stage 2715 then shows the initiation of the switch cameraoperation through the selection of the PIP display area 880 of the UI805. As shown, a selection is made by placing the user's finger 2770 onthe PIP display 880. The third stage 2720 shows the UI 805 that includesa selectable UI item 2775 (e.g., switch camera button 2775) forrequesting a switch between the cameras of the local device 2700 duringthe video conference. The fourth stage 2725 illustrates the UI 805 afterthe user of the local device 2700 selects (e.g., through a single fingertap) the selectable UI item 2775, and after this selection is indicatedthrough the highlighting of the selectable UI item 2775. By selectingthis selectable UI item 2775, the user is directing the device 2700 toswitch from the front camera of the device 2700 to the back camera ofthe device 2700 during the video conference. In other examples where theback camera of the device 2700 is capturing video, the user's selectionof the selectable UI item 2775 directs the device 2700 to switch fromthe back camera of the device 2700 to the front camera of the device2700. After the fourth stage, the video conference manager sendsinstructions to the CIPU and the remote device to start the switchcamera operation.

The last four stages 2730, 2735, 2740, and 2745 of the UI 805 illustratean example of a switch camera animation on the local device. Thisanimation is intended to provide an impression that the video capturedfrom the front and the back cameras of the local device are beingconcurrently displayed on two opposing sides of a viewing pane that canhave only one of its sides viewed by the user at any given time. When aswitch camera is requested in the middle of a video conference, thisviewing pane is made to appear to rotate around the vertical axis suchthat the presentation of one camera's video on one side of the viewingpane that was previously showing one camera's video to the user rotatesaway from the user until it is replaced by the other side of the viewingpane, which shows the video of the other camera. This animation andappearance of the perceived viewing pane's rotation is achieved by (1)gradually shrinking and applying perspective correction operations onthe video image from one camera in the display area for that camera,followed by (2) a gradual expansion and reduction in perspectivecorrection operation to the video image from the other camera in thedisplay area.

Accordingly, the fifth stage 2730 illustrates the start of the “rotationof the viewing pane” about the vertical axis 2782. To give an appearanceof the rotation of the viewing pane, the UI 805 has reduced the size ofthe front camera's video image in the video display area 860, and hasapplied perspective operations to make it appear that the right side ofthe video image is farther from the user than the left side of the videoimage.

The sixth stage 2735 illustrates that the viewing pane has rotated by 90degrees such that the user can only view the edge of this pane, asrepresented by the thin line 2786 displayed in the middle of the displayarea 860. The seventh stage 2740 illustrates that the viewing pane hascontinued to rotate such that the backside of the viewing pane 2788 isnow gradually appearing to the user in order to show the video capturedfrom the user's back camera. Again, this representation of the rotationanimation is achieved in some embodiments by reducing the size of theback camera's video image in the video display area 2788, and applyingperspective operations to make it appear that the left side of the videoimage is farther from the user than the right side of the video image.

The eighth stage 2745 illustrates the completion of the animation thatshows the switch camera operation. Specifically, this stage displays inthe display area 860 the video image of a car that is being captured bythe back camera of the device 2700.

The example described above by reference to FIG. 27 invokes a switchcamera operation through a switch camera user interface. Otherembodiments invoke a switch camera operation differently. For example,some embodiments invoke the switch camera operation by having a switchcamera selectable UI item permanently displayed on a UI during a videoconference such the UI 805 of FIG. 28. In FIG. 28, a switch camerabutton 989 is shown in a display area 855 along with a mute button 985and an end conference button 987. The layout of the display area 855 isthe same layout of the display area 855, described above by reference toFIG. 9.

FIG. 28 illustrates the switch camera operation of a UI 805 in terms ofsix stages: 2710, 2890, 2730, 2735, 2740, and 2745. The first stage 2710of FIG. 28 is similar to the first stage 2710 of FIG. 27 except that thelayout of the display area 855 shows a mute button 985, an endconference button 987, and a switch camera button 989 instead of asingle end conference button. The second stage 2890 illustrates the UI805 after the user of the local device 2700 selects (e.g., through asingle finger tap using a finger 2770) the switch camera selectable UIitem 989. In this example, by selecting this selectable UI item 989, theuser directs the device 2700 to switch from the front camera of thedevice 2700 to the back camera of the device 2700 during the videoconference. The last four stages of FIG. 28 are similar to the last fourstages of FIG. 27 except the layout of the display area 855 is the sameas the layout described above in the first stage 2710 and therefore willnot be further described in order to not obscure the description of theinvention with unnecessary detail.

In some embodiments, when the remote mobile device receives images froma different camera of the local dual camera mobile device (i.e., thelocal dual camera mobile device switched cameras), the remote mobiledevice also performs a switch camera animation to display a transitionbetween the display of image from one camera of the local dual cameramobile device and the display of images from the other camera of thelocal dual camera mobile device. FIG. 29 illustrates an example of oneof such switch camera animation in terms of five operational stages2910, 2915, 2920, 2925, and 2930 of a UI 2905. This figure shows anexample switch camera animation on the remote mobile device 2900. Theoperational stages are the same as the example animation of FIG. 27except the animation is performed on images displayed in the displayarea 2935, which is where images from the local dual camera mobiledevice are displayed on the remote mobile device 2900. As such, theimage of the man displayed in the display area 2935 is animated toappear to rotate 180 degrees on a vertical axis 2955 located in themiddle of the display area 2950 to show the transition between thedisplay of the image of the man in the display area 2935 and the displayof the image of a car 2970. The implementation of the switch cameraanimation of some embodiments is the same as the implementation of theanimation described above.

The above example illustrates a switch camera animation on a remotedevice with a particular user interface layout. Other embodiments mightperform this switch camera animation on a remote device with a differentuser interface layout. For instance, FIG. 30 illustrates one suchexample of a remote device 2900 that has a different user interfacelayout 2905. In particular, UI 2905 of FIG. 30 has a mute button 985, anend conference button 987, and a switch camera button 989 included in adisplay area 855, which is permanently displayed on one side of thecomposite display 2950 during a video conference. The layout of thethree buttons is described above by reference to FIG. 29. Other than thedifferent user interface layout, the five stages 2910, 2915, 2920, 2925,and 2930 of FIG. 30 are identical to the five stages 2910, 2915, 2920,2925, and 2930 of FIG. 29.

2. Remote Switch Camera

FIG. 31 illustrates a process 3100 for switching between two cameras ofa remote dual camera device during a video conference. This process 3100is performed by a video conference manager of a device that includes atleast one camera. In the following discussion, the device through whicha user directs a remote switch camera is referred to as the local devicewhile the device that switches between its two cameras is referred to asthe remote device. Also, in the discussion below, the remote device issaid to switch between its front camera (or camera 1) and its backcamera (or camera 2).

The process 3100 of FIG. 31 will be described by reference to FIGS. 32,33, 34, and 35. FIG. 32 illustrates a UI 3205 of a local device 3200through which a user requests that a remote device switch between itstwo cameras during a video conference. This figure illustrates eightdifferent operational stages 3210, 3215, 3220, 3225, 3230, 3235, 3240,and 3245 of this UI 3205. FIG. 35 illustrates a UI 3505 of a remotedevice 3500 that receives the switch camera request from the localdevice 3200. FIG. 35 illustrates six different operational stages 3510,3515, 3520, 3525, 3530, and 3535 of the UI 3505.

As shown in FIG. 31, the process 3100 begins by starting (at 3105) avideo conference between the local and remote devices. The process 3100then (at 3110) receives images from one camera of each device (e.g.,from the front camera of each device) and generates a composite view forthe video conference based on these images. At 3110, the process 3100also sends a video image from the local device to the remote device.

Next, the process 3100 determines (at 3115) whether a request to end thevideo conference has been received. As described above, a videoconference can end in some embodiments at the request of a user of thelocal or remote device. When the process 3100 receives a request to endthe video conference, the process 3100 ends.

When the process 3100 does not receive a request to end the videoconference, the process 3100 then determines (at 3120) whether the userof the device on which the process 3100 is executing (i.e., the user ofthe local device) has directed the device to request that the remotedevice switch between its cameras for the video conference. The process3100 returns to operation 3110 when the process 3100 determines (at3120) that it has not been directed to initiate a remote switch camera.When the process 3100 determines (at 3120) that it has been so directed,the process 3100 transitions to 3125, which will be described furtherbelow.

The first four stages 3210, 3215, 3220, and 3225 of the UI 3205 of FIG.32 illustrate an example of receiving a user's request to switch camerasof the remote device. The first and second stages 3210 and 3215 are thesame as the first and second stages 2710 and 2715 of FIG. 27. The thirdstage 3220 is the same as the third stage 2720 except the third stage3220 includes a selectable UI item 3280 for a request to the remotedevice 3200 to switch cameras in addition to the selectable UI item 3275for requesting the local device 3200 to switch cameras. The fourth stage3225 illustrates the user of the local device 3200 selecting the UI item3280 (e.g., through a single finger tap 3270 of the selectable UI item3280) for requesting the remote device to switch cameras. The selectionis indicated by the highlighting of the selectable UI item 3280. FIG. 32shows one example of performing this operation, but other embodimentsmay differently perform the operation for requesting the remote deviceto switch cameras.

The example described above by reference to FIG. 32 invokes a remoteswitch camera operation through a remote switch camera user interface.Other embodiments invoke a remote switch camera operation differently.For instance, some embodiments invoke the switch camera operation byhaving a switch camera selectable UI item permanently displayed on a UIduring a video conference such as the UI 3205 of FIG. 33. In FIG. 33, aremote switch camera button 3388 is shown in a display area 855 alongwith a mute button 3382, an end conference button 3384, and a localswitch camera button 3386.

FIG. 33 illustrates the remote switch camera operation of the UI 3205 ofthe device 3200 in terms of six different stages 3210, 3390, 3230, 3235,3240, and 3245. The first stage 3210 of FIG. 33 is similar to the firststage 3210 of FIG. 32 except that the layout of the display area 855shows a mute button 3382, a local switch camera button 3386, a remoteswitch camera button 3388, and an end conference button 3384. The secondstage 3390 illustrates the UI 805 after the user of the local device3200 selects (e.g., through a single finger tap 3270) the remote switchcamera selectable UI item 3388. The last four stages of FIG. 33 aresimilar to the last four stages of FIG. 32 except the layout of thedisplay area 855 is the same as the layout described above in the firststage 3210 and therefore will not be further described in order to notobscure the description of the invention with unnecessary detail.

Some embodiments provide a similar layout as the one illustrated in FIG.33 except the remote switch camera selectable UI item is displayed inPIP display 3265 instead of the display area 855. FIG. 34 illustratessuch a layout 3205. In particular, the figure shows the PIP display withthe remote switch camera selectable UI item 3280 and the display area855 with only a mute button 3382, a local switch camera button 3386, andan end conference button 3384.

As mentioned above, the process 3100 transitions to 3125 when the userrequests a remote switch camera. At 3125, the process 3100 sends therequest to switch cameras to the remote device. In some embodiments,this request is sent through the video conference control channel thatis multiplexed with the audio and video channels by the VTP Manager 1825as described above.

After the request to switch cameras is received, the process 3100determines (at 3130) whether the remote device has responded to therequest to switch cameras. In some embodiments, the remote deviceautomatically sends an accept response (i.e., sends an acknowledgement)to the local device through the video-conference control channel. Inother embodiments, however, the user of the remote device has to acceptthis request through the user interface of the remote device.

The first two stages 3510 and 3515 of the UI 3505 of FIG. 35 illustratean example of the remote user accepting a request to switch cameras ofthe remote device 3500. The first stage 3510 shows (1) a display area3540 for displaying text that notifies the remote user of the request,(2) a selectable UI item 3565 (e.g., allow button 3565) for acceptingthe request to switch cameras of the remote device, and (3) a selectableUI item 3570 (e.g., reject button 3570) for rejecting the request toswitch cameras of the remote device. The second stage 3515 thenillustrates the UI 3505 after the user of the remote device has selected(e.g., through a single finger tap 3580) the UI item 3565 for acceptingthe request to switch cameras, as indicated by the highlighting of theselectable UI item 3565.

When the process 3100 determines (at 3130) that it has not yet receiveda response from the remote device, the process 3100 determines (at 3135)whether a request to end the video conference has been received. If so,the process 3100 returns to operation 3110 to continue to receive imagesfrom the camera of the other device. Otherwise, the process receives (at3140) images from the currently used cameras of the remote and localdevices, generates a composite view for the video conference based onthese images, transmit the local device's video image to the remotedevice, and then transitions back to 3130.

When the process 3100 determines (at 3130) that it has received aresponse from the remote device, it determines (at 3145) whether theremote device accepted the request to switch cameras. If not, theprocess 3100 ends. Otherwise, the process receives (at 3150) images fromthe other camera of the remote device and then performs (at 3155) aswitch camera animation on the local device to display a transitionbetween the video of the previously utilized remote camera and the videoof the currently utilized remote camera (i.e., the received images atoperation 3150). After 3155, the process transitions back to 3110, whichwas described above.

The last four operational stages 3230, 3235, 3240, and 3245 that areillustrated for the UI 3205 in FIG. 32 illustrate one example of such aremote switch camera animation on the local device 3200. The exampleanimation is similar to the example animation illustrated in the stages2915, 2920, 2925, and 2930 of FIG. 29 except FIG. 32 shows in thedisplay area 3250 an animation that replaces the video of a woman thatis captured by the front camera of the remote device with the video of atree that is captured by the back camera of the remote device. The lastfour stages of FIG. 33 and FIG. 34 illustrate the same animation as theone in FIG. 32 except the display area 855 of FIGS. 33 and 34 containsdifferent selectable UI items than the display area 855 in FIG. 32.

In some embodiments, when the remote device switches cameras, the UI ofthe remote device also performs a switch camera animation to display atransition between the two cameras. The last four operational stages3520, 3525, 3530, and 3535 that are illustrated for the UI 3505 in FIG.35 illustrate an example of a switch camera animation that is displayedon the remote device 3500 when the remote device 3500 switches betweencameras. This animation is similar to the animation illustrated in thestages 2730, 2735, 2740, and 2745 of FIG. 27 except that the animationin the display area 3545 replaces the video of a woman that is capturedby the front camera of the remote device 3500 with the video of a treethat is captured by the back camera of the remote device 3500.

As noted above, FIGS. 27, 28, 29, 30, 32, 33, 34, and 35 show variousexamples of switch camera animations performed on a user interface. Insome embodiments, the switch camera animation causes changes to theimage processing operations of the respective dual camera mobile devicesuch as scaling, compositing, and perspective distortion, which can beperformed by the video conference manager 1204 and the image processingmanager 1208, for example.

C. Exposure Adjustment

During a video conference between a dual camera mobile device andanother mobile device, different embodiments provide differenttechniques for adjusting the exposure of images captured by cameras ofeither mobile device. Some embodiments provide techniques for a user ofthe dual camera mobile device to adjust the exposure of images capturedby a camera of the other device while other embodiments providetechniques for the user to adjust the exposure of images captured by acamera of the dual camera mobile device. Several example techniques willbe described in detail below.

FIG. 36 illustrates a process 3600 for performing a remote exposureadjustment operation on a dual camera mobile device of some embodimentsduring a video conference. In the following discussion, the devicethrough which a user directs a remote device to adjust its exposurelevel is referred to as the local device. In some embodiments, theprocess 3600 is performed by the video conference manager of the localdevice. In addition, the process 3600 will be described by reference toFIGS. 37, 38, and 39 which illustrate various ways for the user of thelocal device to request the remote device to perform an exposureadjustment operation.

As shown in FIG. 36, the process 3600 begins by starting (at 3605) avideo conference between the local and remote devices. The process 3600then receives (at 3610) a video from the remote device for display onthe display screen of the local device. Next, the process 3600determines (at 3615) whether a request to end the video conference hasbeen received. As described above, some embodiments can receive arequest to end the video conference from a user of the local or remotedevice. When the process 3600 receives a request to end the videoconference, the process 3600 ends.

However, when the process 3600 does not receive a request to end thevideo conference, the process 3600 then determines (at 3620) whether arequest for adjusting the exposure of the remote device's camera hasbeen received. When the process 3600 determines that a request foradjusting the exposure of the remote device's camera has not beenreceived, the process 3600 returns back to operation 3610 to receiveadditional video captured from the remote device. FIGS. 37, 38, and 39illustrate three different examples of providing a way for a user tomake such a request. In FIGS. 37, 38, and 39, the first stages 3710,3810, and 3910 all show PIP displays 3725, 3850, and 3935 of the localdevices 3700, 3800, and 3900 that display two videos: one captured by acamera of the local device and the other captured by a camera of theremote device. In first stages 3710, 3810, and 3910 the man in thebackground display 3735, 3860, and 3945 is dark, indicating that the manis not properly exposed.

The second stage 3715 of FIG. 37 illustrates one way for the user of thelocal device 3700 to request the remote device to perform an exposureadjustment by selecting the remote device's video (e.g., through asingle tap on the background display 3735). In this way, the UI 3705automatically associates the user's selection of a region of interestdefined by a box 3745 with the user's desire to direct the remote deviceto perform an exposure adjustment on the region of interest and thusdirects the video conference manager of the local device to contact theremote device to perform an exposure adjustment operation. The definedregion of interest is used by the remote device in the calculation ofthe exposure adjustment.

Like the second stage 3715 of FIG. 37, the second stage 3815 of FIG. 38shows the local user's selection of the remote device's video exceptthis selection directs the UI 3805 to display a selectable UI item 3870as shown in the third stage 3820. The fourth stage 3825 illustrates theuser of the local device selecting the selectable UI item 3870 to directthe remote device to perform an exposure adjustment operation asdescribed above.

The second stage 3915 of FIG. 39 is similar to the second stage 3815 ofFIG. 38, but instead of the user's selection of the remote device'svideo directing the UI to display a single selectable UI item, theuser's selection directs the UI 3905 to display a menu of selectable UIitems 3955, 3960, 3965, and 3970, as shown in the third stage 3920. Theselectable UI items include an Auto Focus item 3955, an Auto Exposureitem 3960, a Switch Camera item 3965, and a Cancel item 3970. In someembodiments, the Switch Camera selectable UI item 3965 is used torequest a local switch camera operation while in other embodiments theSwitch Camera selectable UI item 3965 is used to request a remote switchcamera operation. The fourth stage 3925 illustrates the user selectingthe Auto Exposure item 3960 to direct the remote device to perform anexposure adjustment operation as described above.

When the process 3600 determines (at 3620) that the local user directedthe local device to request an exposure adjustment operation, theprocess 3600 sends (at 3625) a command to the remote device through thevideo conference control channel to adjust the exposure of the videocaptured by the camera that is currently capturing and transmittingvideo to the local device. After operation 3625, the process 3600transitions back to operation 3610, which is described above.

In some embodiments, the user of the remote device is required toprovide permission before the remote device performs an exposureadjustment operation, while in other embodiments the remote deviceperforms the exposure adjustment operation automatically upon receivingthe request from the local device. Moreover, in some embodiments, someof the video conference functionalities are implemented by the videoconference manager 1204. In some of these embodiments, the videoconference manager 1204 performs the exposure adjustment operation byinstructing the CIPU 1250 to adjust the exposure setting of the sensorof the remote device camera being used.

The last stages 3720, 3830, and 3930 of FIGS. 37, 38, and 39 show theremote device's video lighter, which indicates that the man is properlyexposed. Although FIGS. 37, 38, and 39 provide examples of receiving anexposure adjustment request to correct the exposure of a remote device,some embodiments provide ways for user of the local device to requestthat the local device adjust the exposure of a camera of the localdevice. Such a request can be made similar to the ways illustrated inFIGS. 37, 38, and 39 for requesting a remote device to adjust itscamera's exposure.

FIGS. 37-39 described above show several user interfaces for performingexposure adjustment operations. In some embodiments, the exposureadjustment operation can cause changes to the image processingoperations of the dual camera mobile device such as invoking theexposure adjustment process 4000, which is described in further detailbelow. The exposure adjustment operation can also cause changes to theoperation of the camera of the dual camera mobile device that iscapturing the video like changing the exposure level setting of thecamera, for example.

FIG. 40 conceptually illustrates an exposure adjustment process 4000performed by an image processing manager of some embodiments such asthat illustrated in FIG. 12. In some embodiments, the process 4000 ispart of the exposure adjustment operations described above by referenceto FIGS. 36, 37, 38, and 39. In some of such embodiments, the imageprocessing manager 1208 performs the process 4000 and adjusts a camera'sexposure setting by sending instructions to the video conference manager1204, which instructs the CIPU 1250 to adjust the camera sensor 405 a or405 b, as mentioned above.

In some embodiments, the process 4000 is performed by the imageprocessing layer 630 shown in FIG. 6 while in other embodiments theprocess 4000 is performed by the statistics engine 465 shown in FIG. 4.Some embodiments perform the process 4000 on images captured by camerasof (local or remote) devices in a video conference while otherembodiments perform the process 4000 as part of the process 1500 (e.g.,operation 1510) illustrated in FIG. 15. Some embodiments perform anexposure adjustment operation to expose images captured by the camerasof the dual camera mobile device that are not too light and not toodark. In other words, the process 4000 is performed to capture images ina manner that maximizes the amount of detail as possible.

The process 4000 begins by receiving (at 4005) an image captured by acamera of the dual camera mobile device. In some embodiments, when thereceived image is a first image captured by a camera of a device in avideo conference, the process 4000 is not performed on the first image(i.e., there was no image before the first image from which to determinean exposure value). The process 4000 then reads (at 4010) pixel valuesof a defined region in the received image. Different embodiments defineregions differently. Some of such embodiments define differently shapedregions such as a square, a rectangle, a triangle, a circle, etc. whileother of such embodiments define regions in different locations in theimage such as center, upper center, lower center, etc.

Next, the process 4000 calculates (at 4015) an average of the pixelvalues in the defined region of the image. The process 4000 determines(at 4020) whether the calculated average of the pixel values is equal toa particular defined value. Different embodiments define differentparticular values. For example, some embodiments define the particularvalue as the median pixel value of the image's dynamic range. In someembodiments, a range of values is defined instead of a single value. Insuch embodiments, the process 4000 determines (at 4020) whether thecalculated average of the pixel values is within the define range ofvalues.

When the calculated average of the pixel values is not equal to theparticular defined value, the process 4000 adjusts (at 4025) theexposure value based on the calculated average. When the calculatedaverage of the pixel values is equal to the particular defined value,the process 4000 ends. In some embodiments, an exposure value representsan amount of time that a camera sensor is exposed to light. In someembodiments, the adjusted exposure value is used to expose the nextimage to be captured by the camera that captured the received image.After the exposure value is adjusted based on the calculated average,the process 4000 ends.

In some embodiments, the process 4000 is repeatedly performed until thecalculated average of pixel values is equal to the particular definedvalue (or falls within the defined range of values). Some embodimentsconstantly perform the process 4000 during a video conference whileother embodiments perform the process 4000 at defined intervals (e.g., 5seconds, 10 seconds, 30 seconds, etc.) during the video conference.Furthermore, during the video conference, the process 4000 of someembodiments dynamically re-defines the particular pixel value beforeperforming the process 4000.

FIG. 41 conceptually illustrates examples of exposure adjustmentoperations of some embodiments. Each of the examples 4100, 4110, and4115 shows an image 4120 captured by a camera of the dual camera mobiledevice on the left side. Specifically, the image 4120 shows a darkperson in front of a sun. The dark person indicates that the exposurelevel of the image is not high enough to expose the person's face orbody. The right side of each example 4100, 4110, and 4115 shows an image4125, 4130, and 4135, respectively, captured after the image 4120. Insome embodiments, the image 4120 and the images on the right side areimages of a video captured by the camera of the dual camera mobiledevice. In other embodiments, the image 4120 and the image on the rightside are still images captured by the camera of the dual camera mobiledevice at different instances in time.

The first example 4100 illustrates an operation with no exposureadjustment. As such, the image 4125 appears the same as the image 4120.Since no exposure adjustment was performed, the person in the image 4125remains dark like the person in the image 4120.

In the second example 4110, an exposure adjustment operation isperformed on the image 4120. In some embodiments, the exposureadjustment operation is performed by the process 4000 using the definedregion 4140. Based on the exposure adjustment operation, the exposurelevel of the camera is adjusted and the camera captures the image 4130using the adjusted exposure level. As shown in FIG. 41, the person inthe image 4130 is not as dark as the in the image 4125. However, theperson's face and body in the image 4130 is still not clear.

The third example 4115 shows an exposure adjustment operation performedon the image 4120. Similar to the second example 4110, the exposureadjustment operation of the example 4115 of some embodiments isperformed by the process 4000 using the defined region 4145. Based onthe exposure adjustment operation, the exposure level of the camera isadjusted and the camera captures the image 4135 using the adjustedexposure level. As seen in FIG. 41, the person in the image 4135 isperfectly exposed since the person's face and body is visible.

In some embodiments, the selection of the defined region may be made bythe user of the dual camera mobile device. The device itself may alsoautomatically adjust its defined region for the exposure adjustmentoperation through the feedback loop for exposure adjustment mentionedabove in the CIPU 400. The statistics engine 465 in FIG. 4 may collectdata to determine whether the exposure level is appropriate for theimages captured and adjust the camera sensors (e.g., though a directconnection to the sensor module 415) accordingly.

D. Focus Adjustment

FIG. 42 illustrates a process 4200 for adjusting the focus of a dualcamera mobile device during a video conference. In the followingdiscussion, the device through which a user directs a remote device toadjust its camera focus is referred to as the local device. The process4200 of FIG. 42 is in some embodiments performed by the video conferencemanager 1204 of the local device. Also, this process will be describedbelow by reference to FIGS. 43 and 44, which provide two exemplarymanners for the user of the local device to request a focus adjustmentoperation to be performed by the remote device.

As shown in FIG. 42, the process 4200 begins by starting (at 4205) avideo conference between the local and remote devices. The process 4200then receives (at 4210) a video from the remote device for display onthe display screen of the local device. Next, at 4215, the process 4200determines whether a request to end the video conference has beenreceived. As described above, a video conference can end in someembodiments at the request of a user of the local or remote device. Whenthe process 4200 receives a request to end the video conference, theprocess 4200 ends.

Otherwise, the process determines (at 4220) whether it has received arequest for adjusting the focus of the remote camera of the remotedevice. When the process 4200 determines that it has not received arequest for adjusting the focus of the remote camera of the remotedevice, the process 4200 returns to operation 4210 to receive additionalvideo from the remote device. FIGS. 43, 44, and 45 illustrate threedifferent ways that different embodiments provide to a user to make sucha request. In FIGS. 43, 44, and 45, the first stages 4310, 4410, and4572 all show a PIP display 4325, 4435, and 4582 of the local device4300, 4400, and 4571 that displays two videos, one captured by the localdevice, and the other captured by the remote device. The display areas855 and 855 in FIGS. 43 and 44 show an end conference button. However,in FIG. 45, the layout of the display area 855 is the same as the layoutof the display area 855 of FIG. 9, described above. Moreover, the switchcamera button 4588 shown in the display area 855 can be selected toinvoke a local switch camera operation in some embodiments or a remoteswitch camera operation in other embodiments. As shown in the firststages 4310, 4410, and 4572, the video of the remote device that isdisplayed in the background display 4335, 4445, and 4580 is blurry.

The second stage 4315 of FIG. 43 illustrates an approach whereby theuser of the local device requests a focus adjustment from the remotedevice by simply selecting the remote device's video (e.g., through asingle tap 4340 on the remote device's video). Under this approach, theUI 4305 automatically associates the user's selection of a region ofinterest defined by a box 4345 with the user's desire to direct theremote device to perform an operation (such as focus) on the region ofinterest and therefore directs the video conference manager 1204 of thelocal device 4300 to contact the remote device to perform an adjustmentoperation (such as an focus adjustment operation). The defined region ofinterest is used by the remote device in the calculation of the focusadjustment.

The second stage 4415 of FIG. 44 similarly shows the local user'sselection of the remote video (e.g., through the user's tapping of theremote device's video). However, unlike the example illustrated in FIG.43, this selection in FIG. 44 directs the UI 4405 to display a menu ofselectable UI items 4455, 4460, 4465 and 4470 (which can be implementedas selectable buttons), as shown in the third stage 4420. Theseselectable UI items include an Auto Focus item 4460, an Auto Exposureitem 4465, a Switch Camera item 4470 and a Cancel item 4455. In someembodiments, the Switch Camera selectable UI item 4470 is used torequest a local switch camera operation while in other embodiments theSwitch Camera selectable UI item 4470 is used to request a remote switchcamera operation. The fourth stage 4425 then illustrates the local userselecting the auto-focus item 4460.

The second stage 4574 of FIG. 45 again similarly shows the local user'sselection of the remote video (e.g., through the user's tapping of theremote device's video). However, unlike the example illustrated in FIG.44, this selection in FIG. 45 directs the UI 4578 to request a focusadjustment operation (i.e., in second stage 4574). After the focusadjustment operation is completed, the UI 4578 displays a menu ofselectable UI items 4584 and 4586 (i.e., in third stage 4576), which canbe implemented as selectable buttons. These selectable UI items includean Auto Exposure item 4586 and a Cancel item 4584.

When the process determines (at 4220) that the local user directed thelocal device to request a focus adjustment operation, the process 4200sends (at 4240) a command to the remote device through the videoconference control channel to adjust the focus of the camera whose videothe remote device is currently capturing and transmitting. After 4240,the process transitions back to 4210, which was described above.

In some embodiments, the user of the remote device has to providepermission before the remote device performs this operation, while inother embodiments the remote device performs this operationautomatically upon receiving the request for the local device. Also, insome embodiments, the focus adjustment operation adjusts the focussettings of the remote device's camera that is being used during thevideo conference. In some of such embodiments, some of the videoconference functionalities are implemented by the video conferencemodule 1202 as discussed above. In these embodiments, the videoconference manager 1204 instructs the CIPU 1250 to adjust the sensor ofthe remote device camera being used.

The last stages 4320, 4430, and 4576 of FIGS. 43, 44, and 45 show theremote device's video properly focused. Although FIGS. 43, 44, and 45provide examples of receiving a focus adjustment request to correct thefocus of a remote device, some embodiments allow the local device's userto request that the local device adjust the focus of a camera of thelocal device. Such a request can be made similar to the approaches shownin FIGS. 43, 44, and 45 to requesting a remote device to adjust itscamera's focus.

FIGS. 43, 44, and 45 illustrate three example user interfaces that allowa user to perform a focus adjustment operation. In some embodiments, thefocus adjustment operation causes changes to the operation of the cameraof the dual camera mobile device that is capturing the video displayedin the UIs such as changing the focus of the camera.

As discussed above in FIGS. 37 and 43, the defined region of interestwas used by the remote mobile device in the computation for exposureadjustment and focus adjustment of the videos, respectively. However, insome other embodiments, the user's selection of a region of interest maybe used to direct the remote device to perform one or more operations.For example, in some embodiments, both exposure adjustment and focusadjustment may be performed based on the defined region of interest,thereby directing the remote device to perform both operations.

E. Frame Rate Control

During a video conference, some embodiments may wish to adjust ormaintain the rate at which images of a video captured by a camera of thedual camera mobile device are transmitted (i.e., frame rate) to theother device in the video conference. For example, assuming a fixedbandwidth, some of such embodiments reduce the frame rate of the videoto increase the picture quality of the images of the video while otherof such embodiments increase the frame rate of the video to smooth outthe video (i.e., reduce jitter).

Different embodiments provide different techniques for controlling theframe rate of images of a video during the video conference. One examplepreviously described above adjusts the VBI of the sensor module 415 fora camera in order to control the rate at which images captured by thecamera are processed. As another example, some embodiments of themanagement layer 635 of the video conference module 625 shown in FIG. 6control the frame rate by dropping images. Similarly, some embodimentsof the image processing layer 630 control the frame rate by droppingimages. Some embodiments provide yet other techniques for controllingframe rates such as dropping frames in the universal transmission buffer1820.

V. Electronic System

Many of the above-described features and applications are implemented assoftware processes that are specified as a set of instructions recordedon a computer readable storage medium (also referred to as computerreadable medium). When these instructions are executed by one or moreprocessing unit(s) (e.g., one or more processors, cores of processors,or other processing units), they cause the processing unit(s) to performthe actions indicated in the instructions. Examples of computer readablemedia include, but are not limited to, CD-ROMs, flash drives, RAM chips,hard drives, EPROMs, etc. The computer readable media does not includecarrier waves and electronic signals passing wirelessly or over wiredconnections.

In this specification, the term “software” is meant to include firmwareresiding in read-only memory or applications stored in magnetic storagewhich can be read into memory for processing by a processor. Also, insome embodiments, multiple software inventions can be implemented assub-parts of a larger program while remaining distinct softwareinventions. In some embodiments, multiple software inventions can alsobe implemented as separate programs. Finally, any combination ofseparate programs that together implement a software invention describedhere is within the scope of the invention. In some embodiments, thesoftware programs, when installed to operate on one or more electronicsystems, define one or more specific machine implementations thatexecute and perform the operations of the software programs.

Some embodiments are implemented as software processes that include oneor more application programming interfaces (APIs) in an environment withcalling program code interacting with other program code being calledthrough the one or more interfaces. Various function calls, messages orother types of invocations, which further may include various kinds ofparameters, can be transferred via the APIs between the calling programand the code being called. In addition, an API may provide the callingprogram code the ability to use data types or classes defined in the APIand implemented in the called program code.

At least certain embodiments include an environment with a callingsoftware component interacting with a called software component throughan API. A method for operating through an API in this environmentincludes transferring one or more function calls, messages, other typesof invocations or parameters via the API.

One or more Application Programming Interfaces (APIs) may be used insome embodiments. For example, some embodiments of the media exchangemodule 310 (or 910) provide a set of APIs to other software componentsfor accessing various video processing and encoding functionalitiesdescribed in FIGS. 3 and 6.

An API is an interface implemented by a program code component orhardware component (hereinafter “API-implementing component”) thatallows a different program code component or hardware component(hereinafter “API-calling component”) to access and use one or morefunctions, methods, procedures, data structures, classes, and/or otherservices provided by the API-implementing component. An API can defineone or more parameters that are passed between the API-calling componentand the API-implementing component.

An API allows a developer of an API-calling component (which may be athird party developer) to leverage specified features provided by anAPI-implementing component. There may be one API-calling component orthere may be more than one such component. An API can be a source codeinterface that a computer system or program library provides in order tosupport requests for services from an application. An operating system(OS) can have multiple APIs to allow applications running on the OS tocall one or more of those APIs, and a service (such as a programlibrary) can have multiple APIs to allow an application that uses theservice to call one or more of those APIs. An API can be specified interms of a programming language that can be interpreted or compiled whenan application is built.

In some embodiments the API-implementing component may provide more thanone API, each providing a different view of or with different aspectsthat access different aspects of the functionality implemented by theAPI-implementing component. For example, one API of an API-implementingcomponent can provide a first set of functions and can be exposed tothird party developers, and another API of the API-implementingcomponent can be hidden (not exposed) and provide a subset of the firstset of functions and also provide another set of functions, such astesting or debugging functions which are not in the first set offunctions. In other embodiments the API-implementing component mayitself call one or more other components via an underlying API and thusbe both an API-calling component and an API-implementing component.

An API defines the language and parameters that API-calling componentsuse when accessing and using specified features of the API-implementingcomponent. For example, an API-calling component accesses the specifiedfeatures of the API-implementing component through one or more API callsor invocations (embodied for example by function or method calls)exposed by the API and passes data and control information usingparameters via the API calls or invocations. The API-implementingcomponent may return a value through the API in response to an API callfrom an API-calling component. While the API defines the syntax andresult of an API call (e.g., how to invoke the API call and what the APIcall does), the API may not reveal how the API call accomplishes thefunction specified by the API call. Various API calls are transferredvia the one or more application programming interfaces between thecalling (API-calling component) and an API-implementing component.Transferring the API calls may include issuing, initiating, invoking,calling, receiving, returning, or responding to the function calls ormessages; in other words, transferring can describe actions by either ofthe API-calling component or the API-implementing component. Thefunction calls or other invocations of the API may send or receive oneor more parameters through a parameter list or other structure. Aparameter can be a constant, key, data structure, object, object class,variable, data type, pointer, array, list or a pointer to a function ormethod or another way to reference a data or other item to be passed viathe API.

Furthermore, data types or classes may be provided by the API andimplemented by the API-implementing component. Thus, the API-callingcomponent may declare variables, use pointers to, use or instantiateconstant values of such types or classes by using definitions providedin the API.

Generally, an API can be used to access a service or data provided bythe API-implementing component or to initiate performance of anoperation or computation provided by the API-implementing component. Byway of example, the API-implementing component and the API-callingcomponent may each be any one of an operating system, a library, adevice driver, an API, an application program, or other module (itshould be understood that the API-implementing component and theAPI-calling component may be the same or different type of module fromeach other). API-implementing components may in some cases be embodiedat least in part in firmware, microcode, or other hardware logic. Insome embodiments, an API may allow a client program to use the servicesprovided by a Software Development Kit (SDK) library. In otherembodiments an application or other client program may use an APIprovided by an Application Framework. In these embodiments theapplication or client program may incorporate calls to functions ormethods provided by the SDK and provided by the API or use data types orobjects defined in the SDK and provided by the API. An ApplicationFramework may in these embodiments provide a main event loop for aprogram that responds to various events defined by the Framework. TheAPI allows the application to specify the events and the responses tothe events using the Application Framework. In some implementations, anAPI call can report to an application the capabilities or state of ahardware device, including those related to aspects such as inputcapabilities and state, output capabilities and state, processingcapability, power state, storage capacity and state, communicationscapability, etc., and the API may be implemented in part by firmware,microcode, or other low level logic that executes in part on thehardware component.

The API-calling component may be a local component (i.e., on the samedata processing system as the API-implementing component) or a remotecomponent (i.e., on a different data processing system from theAPI-implementing component) that communicates with the API-implementingcomponent through the API over a network. It should be understood thatan API-implementing component may also act as an API-calling component(i.e., it may make API calls to an API exposed by a differentAPI-implementing component) and an API-calling component may also act asan API-implementing component by implementing an API that is exposed toa different API-calling component.

The API may allow multiple API-calling components written in differentprogramming languages to communicate with the API-implementing component(thus the API may include features for translating calls and returnsbetween the API-implementing component and the API-calling component);however the API may be implemented in terms of a specific programminglanguage. An API-calling component can, in one embodiment, call APIsfrom different providers such as a set of APIs from an OS provider andanother set of APIs from a plug-in provider and another set of APIs fromanother provider (e.g. the provider of a software library) or creator ofthe another set of APIs.

FIG. 46 is a block diagram illustrating an exemplary API architecture,which may be used in some embodiments of the invention. As shown in FIG.46, the API architecture 4600 includes the API-implementing component4610 (e.g., an operating system, a library, a device driver, an API, anapplication program, software or other module) that implements the API4620. The API 4620 specifies one or more functions, methods, classes,objects, protocols, data structures, formats and/or other features ofthe API-implementing component that may be used by the API-callingcomponent 4630. The API 4620 can specify at least one calling conventionthat specifies how a function in the API-implementing component 4630receives parameters from the API-calling component 4630 and how thefunction returns a result to the API-calling component. The API-callingcomponent 4630 (e.g., an operating system, a library, a device driver,an API, an application program, software or other module), makes APIcalls through the API 4620 to access and use the features of theAPI-implementing component 4610 that are specified by the API 4620. TheAPI-implementing component 4610 may return a value through the API 4620to the API-calling component 4630 in response to an API call.

It will be appreciated that the API-implementing component 4610 mayinclude additional functions, methods, classes, data structures, and/orother features that are not specified through the API 4620 and are notavailable to the API-calling component 4630. It should be understoodthat the API-calling component 4630 may be on the same system as theAPI-implementing component 4610 or may be located remotely and accessesthe API-implementing component 4610 using the API 4620 over a network.While FIG. 46 illustrates a single API-calling component 4630interacting with the API 4620, it should be understood that otherAPI-calling components, which may be written in different languages (orthe same language) than the API-calling component 4630, may use the API4620.

The API-implementing component 4610, the API 4620, and the API-callingcomponent 4630 may be stored in a machine-readable medium, whichincludes any mechanism for storing information in a form readable by amachine (e.g., a computer or other data processing system). For example,a machine-readable medium includes magnetic disks, optical disks, randomaccess memory; read only memory, flash memory devices, etc.

FIG. 47 is an example of a dual camera mobile computing devicearchitecture 4700. The implementation of a mobile computing device caninclude one or more processing units 4705, memory interface 4710 and aperipherals interface 4715. Each of these components that make up thecomputing device architecture can be separate components or integratedin one or more integrated circuits. These various components can also becoupled together by one or more communication buses or signal lines.

The peripherals interface 4715 can be coupled to various sensors andsubsystems, including a camera subsystem 4720, a wireless communicationsubsystem(s) 4725, audio subsystem 4730, I/O subsystem 4735, etc. Theperipherals interface 4715 enables communication between processors andperipherals. Peripherals such as an orientation sensor 4745 or anacceleration sensor 4750 can be coupled to the peripherals interface4715 to facilitate the orientation and acceleration functions.

The camera subsystem 4720 can be coupled to one or more optical sensors4740, e.g., a charged coupled device (CCD) optical sensor, acomplementary metal-oxide-semiconductor (CMOS) optical sensor. Thecamera subsystem 4720 coupled with the sensors may facilitate camerafunctions, such as image and/or video data capturing. Wirelesscommunication subsystems 4725 may serve to facilitate communicationfunctions. Wireless communication subsystems 4725 may include radiofrequency receivers and transmitters, and optical receivers andtransmitters. They may be implemented to operate over one or morecommunication networks such as a GSM network, a Wi-Fi network, Bluetoothnetwork, etc. The audio subsystems 4730 is coupled to a speaker and amicrophone to facilitate voice-enabled functions, such as voicerecognition, digital recording, etc.

I/O subsystem 4735 involves the transfer between input/output peripheraldevices, such as a display, a touch screen, etc., and the data bus ofthe CPU through the Peripherals Interface. I/O subsystem 4735 caninclude a touch-screen controller 4755 and other input controllers 4760to facilitate these functions. Touch-screen controller 4755 can becoupled to the touch screen 4765 and detect contact and movement on thescreen using any of multiple touch sensitivity technologies. Other inputcontrollers 4760 can be coupled to other input/control devices, such asone or more buttons.

Memory interface 4710 can be coupled to memory 4770, which can includehigh-speed random access memory and/or non-volatile memory such as flashmemory. Memory can store an operating system (OS) 4772. The OS 4772 caninclude instructions for handling basic system services and forperforming hardware dependent tasks.

Memory can also include communication instructions 4774 to facilitatecommunicating with one or more additional devices; graphical userinterface instructions 4776 to facilitate graphic user interfaceprocessing; image/video processing instructions 4778 to facilitateimage/video-related processing and functions; phone instructions 4780 tofacilitate phone-related processes and functions; media exchange andprocessing instructions 4782 to facilitate media communication andprocessing-related processes and functions; camera instructions 4784 tofacilitate camera-related processes and functions; and videoconferencing instructions 4786 to facilitate video conferencingprocesses and functions. The above identified instructions need not beimplemented as separate software programs or modules. Various functionsof mobile computing device can be implemented in hardware and/or insoftware, including in one or more signal processing and/or applicationspecific integrated circuits.

The above-described embodiments may include touch I/O device 4801 thatcan receive touch input for interacting with computing system 4803, asshown in FIG. 48, via wired or wireless communication channel 4802.Touch I/O device 4801 may be used to provide user input to computingsystem 4803 in lieu of or in combination with other input devices suchas a keyboard, mouse, etc. One or more touch I/O devices 4801 may beused for providing user input to computing system 4803. Touch I/O device4801 may be an integral part of computing system 4803 (e.g., touchscreen on a laptop) or may be separate from computing system 4803.

Touch I/O device 4801 may include a touch sensitive panel which iswholly or partially transparent, semitransparent, non-transparent,opaque or any combination thereof. Touch I/O device 4801 may be embodiedas a touch screen, touch pad, a touch screen functioning as a touch pad(e.g., a touch screen replacing the touchpad of a laptop), a touchscreen or touchpad combined or incorporated with any other input device(e.g., a touch screen or touchpad disposed on a keyboard) or anymulti-dimensional object having a touch sensitive surface for receivingtouch input.

In one example, touch I/O device 4801 embodied as a touch screen mayinclude a transparent and/or semitransparent touch sensitive panelpartially or wholly positioned over at least a portion of a display.According to this embodiment, touch I/O device 4801 functions to displaygraphical data transmitted from computing system 4803 (and/or anothersource) and also functions to receive user input. In other embodiments,touch I/O device 4801 may be embodied as an integrated touch screenwhere touch sensitive components/devices are integral with displaycomponents/devices. In still other embodiments a touch screen may beused as a supplemental or additional display screen for displayingsupplemental or the same graphical data as a primary display andreceiving touch input.

Touch I/O device 4801 may be configured to detect the location of one ormore touches or near touches on device 4801 based on capacitive,resistive, optical, acoustic, inductive, mechanical, chemicalmeasurements, or any phenomena that can be measured with respect to theoccurrences of the one or more touches or near touches in proximity todevice 4801. Software, hardware, firmware or any combination thereof maybe used to process the measurements of the detected touches to identifyand track one or more gestures. A gesture may correspond to stationaryor non-stationary, single or multiple, touches or near touches on touchI/O device 4801. A gesture may be performed by moving one or morefingers or other objects in a particular manner on touch I/O device 4801such as tapping, pressing, rocking, scrubbing, twisting, changingorientation, pressing with varying pressure and the like at essentiallythe same time, contiguously, or consecutively. A gesture may becharacterized by, but is not limited to a pinching, sliding, swiping,rotating, flexing, dragging, or tapping motion between or with any otherfinger or fingers. A single gesture may be performed with one or morehands, by one or more users, or any combination thereof.

Computing system 4803 may drive a display with graphical data to displaya graphical user interface (GUI). The GUI may be configured to receivetouch input via touch I/O device 4801. Embodied as a touch screen, touchI/O device 4801 may display the GUI. Alternatively, the GUI may bedisplayed on a display separate from touch I/O device 4801. The GUI mayinclude graphical elements displayed at particular locations within theinterface. Graphical elements may include but are not limited to avariety of displayed virtual input devices including virtual scrollwheels, a virtual keyboard, virtual knobs, virtual buttons, any virtualUI, and the like. A user may perform gestures at one or more particularlocations on touch I/O device 4801 which may be associated with thegraphical elements of the GUI. In other embodiments, the user mayperform gestures at one or more locations that are independent of thelocations of graphical elements of the GUI. Gestures performed on touchI/O device 4801 may directly or indirectly manipulate, control, modify,move, actuate, initiate or generally affect graphical elements such ascursors, icons, media files, lists, text, all or portions of images, orthe like within the GUI. For instance, in the case of a touch screen, auser may directly interact with a graphical element by performing agesture over the graphical element on the touch screen. Alternatively, atouch pad generally provides indirect interaction. Gestures may alsoaffect non-displayed GUI elements (e.g., causing user interfaces toappear) or may affect other actions within computing system 4803 (e.g.,affect a state or mode of a GUI, application, or operating system).Gestures may or may not be performed on touch I/O device 4801 inconjunction with a displayed cursor. For instance, in the case in whichgestures are performed on a touchpad, a cursor (or pointer) may bedisplayed on a display screen or touch screen and the cursor may becontrolled via touch input on the touchpad to interact with graphicalobjects on the display screen. In other embodiments in which gesturesare performed directly on a touch screen, a user may interact directlywith objects on the touch screen, with or without a cursor or pointerbeing displayed on the touch screen.

Feedback may be provided to the user via communication channel 4802 inresponse to or based on the touch or near touches on touch I/O device4801. Feedback may be transmitted optically, mechanically, electrically,olfactory, acoustically, or the like or any combination thereof and in avariable or non-variable manner.

These functions described above can be implemented in digital electroniccircuitry, in computer software, firmware or hardware. The techniquescan be implemented using one or more computer program products.Programmable processors and computers can be included in or packaged asmobile devices. The processes and logic flows may be performed by one ormore programmable processors and by one or more programmable logiccircuitry. General and special purpose computing devices and storagedevices can be interconnected through communication networks.

Some embodiments include electronic components, such as microprocessors,storage and memory that store computer program instructions in amachine-readable or computer-readable medium (alternatively referred toas computer-readable storage media, machine-readable media, ormachine-readable storage media). Some examples of such computer-readablemedia include RAM, ROM, read-only compact discs (CD-ROM), recordablecompact discs (CD-R), rewritable compact discs (CD-RW), read-onlydigital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a varietyof recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.),flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.),magnetic and/or solid state hard drives, read-only and recordableBlu-Ray® discs, ultra density optical discs, any other optical ormagnetic media, and floppy disks. The computer-readable media may storea computer program that is executable by at least one processing unitand includes sets of instructions for performing various operations.Examples of computer programs or computer code include machine code,such as is produced by a compiler, and files including higher-level codethat are executed by a computer, an electronic component, or amicroprocessor using an interpreter.

While the above discussion primarily refers to microprocessor ormulti-core processors that execute software, some embodiments areperformed by one or more integrated circuits, such as applicationspecific integrated circuits (ASICs) or field programmable gate arrays(FPGAs). In some embodiments, such integrated circuits executeinstructions that are stored on the circuit itself.

As used in this specification and any claims of this application, theterms “computer”, “server”, “processor”, and “memory” all refer toelectronic or other technological devices. These terms exclude people orgroups of people. For the purposes of the specification, the termsdisplay or displaying means displaying on an electronic device. As usedin this specification and any claims of this application, the terms“computer readable medium” and “computer readable media” are entirelyrestricted to tangible, physical objects that store information in aform that is readable by a computer. These terms exclude any wirelesssignals, wired download signals, and any other ephemeral signals.

FIG. 49 conceptually illustrates an example communication system 4900used for connecting some participants of a video conference according tosome embodiments. As shown, the communication system 4900 includesseveral mobile devices 4915, several cellular base stations (or Node Bs)4910, several radio network controllers (RNCs) 4905, and a core network4925. Cellular base stations and RNCs are collectively referred to as aUniversal Mobile Telecommunications System (UMTS) Terrestrial RadioAccess Network (UTRAN) 4930. Each RNC 4905 is connected to one or morecellular base stations 4910 that, together, are referred to as a radioaccess network (RAN).

Each cellular base station 4910 covers a service region 4920. As shown,the mobile devices 4915 in each service region are wirelessly connectedto the serving cellular base station 4910 of the service region 4920through a Uu interface. The Uu interface uses a protocol stack that hastwo planes: a control plane and a user plane. The user plane supportscircuit-switched, packet-switched and broadcast data streams. Thecontrol plane carries the network's signaling messages.

Each cellular base station is connected to an RNC through an Iubinterface. Each RNC 4905 is connected to the core network 4925 by Iu-csand an Iu-ps interfaces. The Iu-cs interface is used for circuitswitched services (e.g., voice) while the Iu-ps interface is used forpacket switched services (e.g., data). The Iur interface is used forconnecting two RNCs together.

Accordingly, the communication system 4900 supports bothcircuit-switched services and packet-switched services. For example,circuit-switched services allow a telephone call to be conducted bytransmitting the telephone call data (e.g., voice) throughcircuit-switched equipment of the communication system 4900.Packet-switched services allow a video conference to be conducted byusing a transport protocol layer such as UDP or TCP over an internetlayer protocol like IP to transmit video conference data throughpacket-switched equipment of the communication system 4900. In someembodiments, the telephone call to video conference transition (e.g.,handoff) previously described in the Video Conference Setup section usesthe circuit-switched and packet-switched services supported by acommunication system like the communication system 4900. That is, insuch embodiments, the telephone call is conducted through thecircuit-switched equipment of the communication system 4900 and thevideo conference it conducted through the packet-switched equipment ofthe communication system 4900.

Although the example communication system in FIG. 49 illustrates a thirdgeneration (3G) technology UTRAN wireless mobile communication system,it should be noted that second generation (2G) communication systems,other 3 G communication systems such as 3GPP2 Evolution-Data Optimizedor Evolution-Data only (EV-DO) and 3rd generation partnership project 2(3GPP2) Code Division Multiple Access 1X (CDMA 1X), fourth generation(4G) communication systems, wireless local area network (WLAN), andWorldwide Interoperability for Microwave Access (WiMAX) communicationsystems can be used for connecting some of the participants of aconference in some embodiments. Examples of 2G systems include GlobalSystem for Mobile communications (GSM), General Packet Radio Service(GPRS), and Enhanced Data Rates for GSM Evolution (EDGE). A 2 Gcommunication system architecture is similar to the architecture shownin FIG. 49 except the 2 G communication system architecture uses basetransceiver stations (BTSs) instead of Node Bs 4910 and base stationcontrollers (BSC) instead of RNC 4905. In a 2 G communication system, anA interface between the BSC and the core network is used for circuitswitched services and a Gb interface between the BSC and the corenetwork is used for packet switched services.

In some embodiments, the communication system 4900 is operated by aservice carrier who initially provisions a mobile device 4915 to allowthe mobile device 4915 to use the communication system 4900. Someembodiments provision a mobile device 4915 by configuring andregistering a subscriber identity module (SIM) card in the mobile device4915. In other embodiments, the mobile device 4915 is instead configuredand registered using the mobile device 4915's memory. Moreover,additional services can be provisioned (after a customer purchases themobile device 4915) such as data services like GPRS, multimediamessaging service (MMS), and instant messaging. Once provisioned, themobile device 4915 is activated and is thereby allowed to use thecommunication system 4900 by the service carrier.

The communication system 4900 is a private communication network in someembodiments. In such embodiments, the mobile devices 4915 cancommunicate (e.g., conduct voice calls, exchange data) among each other(e.g., mobile devices 4915 that are provisioned for the communicationsystem 4900). In other embodiments, the communication system 4900 is apublic communication network. Thus, the mobile devices 4915 cancommunicate with other devices outside of the communication system 4900in addition to the mobile devices 4915 provisioned for the communicationsystem 4900. Some of the other devices outside of the communicationsystem 4900 include phones, computers, and other devices that connect tothe communication system 4900 through other networks such as a publicswitched telephone network or another wireless communication network.

The Long-Term Evolution (LTE) specification is used to define 4 Gcommunication systems. FIG. 50 conceptually illustrates an example of a4 G communication system 5000 that is used for connecting someparticipants of a video conference in some embodiments. As shown, thecommunication system 5000 includes several mobile devices 4915, severalEvolved Node Bs (eNBs) 5005, a Mobility Management Entity (MME) 5015, aServing Gateway (S-GW) 5020, a Packet Data Network (PDN) Gateway 5025,and a Home Subscriber Server (HSS) 5035. In some embodiments, thecommunication system 5000 includes one or more MMEs 5015, one or moreS-GWs 5020, one or more PDN Gateways 5025, and one or more HS Ss 5035.

The eNBs 5005 provide an air interface for the mobile devices 4915. Asshown, each eNB 5005 covers a service region 5010. The mobile devices4915 in each service region 5010 are wirelessly connected to the eNB5005 of the service region 5010 through a LTE-Uu interface. FIG. 50 alsoshows the eNBs 5005 connected to each other through an X2 interface. Inaddition, the eNBs 5005 are connected to the MME 5015 through an S1-MMEinterface and to the S-GW 5020 through an S1-U interface. The eNBs 5005are collectively referred to as an Evolved UTRAN (E-TRAN) 5030.

The eNBs 5005 provide functions such as radio resource management (e.g.,radio bearer control, connection mobility control, etc.), routing ofuser plane data towards the S-GW 5020, signal measurement andmeasurement reporting, MME selection at the time of mobile deviceattachment, etc. The MME 5015 functions include idle mode mobile devicetracking and paging, activation and deactivation of radio bearers,selection of the S-GW 5020 at the time of mobile device attachment,Non-Access Stratum (NAS) signaling termination, user authentication byinteracting with the HSS 5035, etc.

The S-GW 5020 functions includes (1) routing and forwarding user datapackets and (2) managing and storing mobile device contexts such asparameters of the IP bearer service and network internal routinginformation. The PDN Gateway 5025 functions include providingconnectivity from the mobile devices to external packet data networks(not shown) by being the point of exit and entry of traffic for themobile devices. A mobile station may have simultaneous connectivity withmore than one PDN Gateway for accessing multiple packet data networks.The PDN Gateway 5025 also acts as the anchor for mobility between 3GPPand non-3GPP technologies such as WiMAX and 3GPP2 (e.g., CDMA 1X andEV-DO).

As shown, MME 5015 is connected to S-GW 5020 through an S11 interfaceand to the HSS 5035 through an S6a interface. The S-GW 5020 and the PDNGateway 5020 are connected through an S8 interface. The MME 5015, S-GW5020, and PDN Gateway 5025 are collectively referred to as an EvolvedPacket Core (EPC). The EPC is the main component of a SystemArchitecture Evolution (SAE) architecture, which is the core networkarchitecture of 3GPP LTE wireless communication standard. The EPC is apure packet system. For example, the EPC does not have a voice mediagateway. Services, like voice and SMS, are packet-switched routed andare provided by application functions that make use of the EPC service.So using the telephone call to video conference transition previouslydescribed above as an example, both the telephone call and the videoconference are conducted through packet-switched equipment of thecommunication system 5000 in some embodiments. In some such embodiments,the packet-switched channel used for the telephone call is continued tobe used for the audio data of the video conference after the telephonecall terminates. However, in other such embodiments, a differentpacket-switched channel is created (e.g., when the video conference isestablished) and audio data is transmitted through the newly createdpacket-switched channel instead of the packet-switched channel of thetelephone call when the telephone call terminates.

Moreover, the amount of bandwidth provided by these differenttechnologies ranges from 44 kilobits per second (kbps) for GPRS to over10 megabits per second (Mbps) for LTE. Download rates of 100 Mbps andupload rates of 50 Mbps are predicted in the future for LTE.

While the invention has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe invention can be embodied in other specific forms without departingfrom the spirit of the invention. In addition, a number of the figuresconceptually illustrate processes. The specific operations of theseprocesses may not be performed in the exact order shown and described.The specific operations may not be performed in one continuous series ofoperations, and different specific operations may be performed indifferent embodiments. Furthermore, the process could be implementedusing several sub-processes, or as part of a larger macro process.

Also, many embodiments were described above by reference to a videoconference between two dual camera mobile devices. However, one ofordinary skill in the art will realize that many of these embodimentsare used in cases involving a video conference between a dual cameramobile device and another device, such as a single camera mobile device,a computer, a phone with video conference capability, etc. Moreover,many of the embodiments described above can be used in single cameramobile devices and other computing devices with video conferencecapabilities. Thus, one of ordinary skill in the art would understandthat the invention is not limited by the foregoing illustrative details,but rather is to be defined by the appended claims.

1. A non-transitory computer readable medium of a first mobile device,the computer readable medium storing a computer program for allowing thefirst device that is in a video conference with a second mobile deviceto remotely control the second mobile device, the computer programcomprising sets of instructions for: sending images captured by a cameraof the first device to the second device; receiving images captured by acamera of the second device; and sending a command through acommunication channel of a real-time communication session to the seconddevice, said command for instructing the second device to perform anoperation that modifies the images captured by the camera of the seconddevice.
 2. The computer readable medium of claim 1, wherein thecommunication channel is a control channel.
 3. The computer readablemedium of claim 1, wherein the operation is an exposure operation. 4.The computer readable medium of claim 1, wherein the operation is afocus operation.
 5. The computer readable medium of claim 1, wherein theoperation is a zoom operation.
 6. The computer readable medium of claim1, wherein the computer program further comprises a set of instructionsfor displaying the images captured by the camera of the second device.7. The computer readable medium of claim 1, wherein the command isreceived through a graphical user interface of the first device.
 8. Thecomputer readable medium of claim 7, wherein the command is sent bytouching a location on a display screen of the first device, thelocation being the location that displays the images captured by thesecond device's camera.
 9. A computer readable medium of a first mobiledevice, the computer readable medium storing a computer program forallowing the first mobile device during a video conference with a secondmobile device to remotely control the second device, the computerprogram comprising a graphical user interface (“GUI”), the GUIcomprising: a first display area for displaying images captured by acamera of the first device during a real-time communication session; asecond display area for displaying images captured by a camera of thesecond device during the real-time communication session; and a userinterface (“UI”) item for sending a command to the second device toinstruct the second device to perform an operation during the real-timecommunication session, said operation modifying the images captured bythe camera of the second device.
 10. The computer readable medium ofclaim 9, wherein the operation is an exposure operation.
 11. Thecomputer readable medium of claim 9, wherein the operation is a focusoperation.
 12. The computer readable medium of claim 9, wherein theoperation is a zoom operation.
 13. The computer readable medium of claim9, wherein the command is sent through a communication channel of thereal-time communication session.
 14. The computer readable medium ofclaim 13, wherein the communication channel is a control channel of thereal-time communication session.
 15. The computer readable medium ofclaim 9, wherein the first display area is inset in the second displayarea.
 16. A method of defining a computer program for allowing a firstmobile device to remotely control a second mobile device during a videoconference, the method comprising: defining a display for displaying avideo captured from the second mobile device on the first mobile device;and defining a camera adjustment operation for receiving an input on thefirst mobile device to direct the second mobile device to perform anoperation during the video conference to modify the images captured bythe camera of the second device.
 17. The method of claim 16 furthercomprising defining a first network manager for the first mobile devicethat relays the received input to a second network manager of the secondmobile device during the video conference.
 18. The method of claim 17,wherein the first network manager relays data to the second networkmanager during the video conference through packets transmitted over theInternet Protocol (IP), said packets including control packets forrelaying input data received on the first mobile device for controllingthe operation of the second mobile device.
 19. The method of claim 16further comprising defining a user interface for receiving the input bydetecting that a location on the display has been contacted during thevideo conference.
 20. The method of claim 16, wherein the operation isan exposure operation.